Amending dayfile: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search
m (Resequence so MX before legacy)
Line 1: Line 1:
{{Template:WorkInProgressBanner}}{{Template:Version badge Mx}}[[Category:Cumulus MX]][[Category:Cumulus Files]][[Category:MX txt Files]][[Category:Cumulus 1]]
Cumulus uses a daily summary log file, the fields in that file are listed at [[Dayfile.txt#List_of_Fields]].  The information about amending the file that was originally on the same page has been moved to this page.


<div style="background: LemonChiffon;padding:5px; margin:2px;">
This page was created with text for the (legacy) Cumulus 1 software, but it has been amended to cover MX as well.
[[File:Crystal Clear info.png|40px]] This text was written for the (legacy) Cumulus 1 software.  It has been updated to cover MX, but that was for a MX release that is no longer latest!
{{Version badge 1}}{{Template:Version badge Mx}}[[Category:Cumulus Files]][[Category:MX txt Files]]


<big>Mark Crossley has now written a utility (download at https://github.com/cumulusmx/CreateMissing) that can update dayfile.txt.</big>
=How Cumulus uses the daily summary log=
</div>


= Safely editing the daily summary log =
While Cumulus is reading your weather station, all daily extremes, counts, and totals, Cumulus needs are stored in [[today.ini]].


At the end of each meteorological day, Cumulus uses information held in [[today.ini]] to write a new line into [[dayfile.txt]], before it resets '''today.ini''' ready for the new day. As discussed in [[Correcting_Extremes]], it is possible for rogue values to be read from a weather station and propagate into various log files, so this article will help you make any necessary corrections.
==Reading the daily summary log file==


The fields contained in the daily summary log are listed in the [[Dayfile.txt#List_of_Fields]]. That page contains a lot of information, and to avoid confusion information about editing this log has been moved here. Both Cumulus 1 and 3 (2 did not) now provide editors where you can see what is in your dayfile.txt log, and if you click on a particular line you can edit the fields in that line, or indeed delete that line. The functionality available to you, depends on the flavour of Cumulus you are running (and which release of that flavour you have installed).
For MX release 3.9.2 - build 3097 onwards, when the software first starts, the whole of '''dayfile.txt''' is read, the contents are used to drive the [[Highcharts_-_Historic]] functionality.


The most common problems with dayfile.txt are:
The file is also read, if you are using the editor provided in Cumulus.
*(MX only) inconsistencies between lines i.e. one of following changes from one line to another
** either the date separator (character or characters between day of month, month, and year parts)
** or the decimal point or decimal comma (that separates the integer and decimal parts of real numbers)
*people see Cumulus giving error messages saying there is an error in a particular line (Cumulus only reports first error, there may be more)
*plots only show data for a subset of the lines in the file, because different lines have different formats for the dates
*a Cumulus user (or an electrical fault) stops or starts the software close to rollover time, a Cumulus user upgrades the software close to rollover time, or another process interferes with Cumulus causing it to twice copy information from '''today.ini''' to '''dayfile.txt''' resulting in a duplicate line with same date (but incorrect content in one of the lines)


All errors generally relate to  
==Writing a new line to the daily summary log file==
* Either using the same file, but moving between computers, without checking the locale settings
 
*Or a mistake in the way a Cumulus user has edited the file, perhaps because of editing the file using Microsoft Excel (and formatting the date column as a date not as text)!  
At the end of each meteorological day, Cumulus uses information held in '''today.ini''' to write a new line into [[dayfile.txt]], before it resets '''today.ini''' ready for the new day.
 
Chill Hours Daily Increment for storing in '''dayfile.txt''' is calculated by substracting the cumulative count in [[yesterday.ini]] from the cumulative count in '''today.ini'''.
 
If rollover, is not at midnight, sunshine hours (these run from midnight to midnight) for the '''dayfile.txt'' are read from '''yesterday.ini'''. For rollover at midnight, sunshine hours from '''today.ini'' (before reset) is read.
 
 
= Problems that occur in the daily summary log and how to safely rectify them =
 
As discussed in [[Correcting_Extremes]], it is possible for rogue values to be read from a weather station, and propagate into various log files. An error in many of those files, corrupts a particular extreme record (or more than one), but generally does not stop Cumulus working.
 
Since MX now reads the entire daily summary log when you start it up, an error in '''dayfile.txt''' can cause a more significant problem, most likely for historic charts, but it can affect other functionality!
 
 
==Summary for most common problems==
 
{| class="wikitable" border="1"
|-
!style="width:100px" | Problem
!style="width:100px" | Possible cause
!style="width:300px" | Rectification for MX releases
!style="width:300px" | Rectification for legacy versions
|-
|  One, or more, past line(s) accidentally deleted or corrupted
| A mistake during a manual view/edit to the file, or a mistake while using the built-in editor
| See [[#Missing or Corrupted past ''dayfile.txt'' lines in any Cumulus software]]
| See [[#Missing or Corrupted past ''dayfile.txt'' lines in any Cumulus software]]
|-
| Inconsistencies in date formats (what comes between day of month, month, and year)
| Examples:
# File was previously used with Cumulus 1 (that did not care what symbol came between parts of date)
# Moving from one device to another (and not ensuring same locale on both devices)
# Editing using an external app (like Excel) and treating date cells as date type, should be text type
| Cannot use editor in MX admin interface, for bulk line edits see [[#Correcting date separator errors]]
| Type over individual dates in built-in editor,  for bulk line edits see [[#Correcting date separator errors]]
|-
| Inconsistencies between decimal comma and decimal points
| Moving from one device to another (and not ensuring same locale on both devices)
| Correct "locale" and/or use an external editor that offers "Replace all" (see [[#Validation by in-built editors]])
| Use '''Control Panel''' to correct region. Correct "locale" and/or use an external editor that offers "Replace all"
|-
| Inconsistencies in list separator (what comes between fields)
| Moving from one device to another (and not ensuring same locale on both devices)
| Correct "locale" and/or use an external editor that offers "Replace all"
| Use '''Control Panel''' to correct region. See [[#Using the Cumulus 1 editing feature]] to check file is now consistent.
|-
| Duplication of dates between lines (either consecutive lines, or non-adjacient lines)
| Electrical fault (or manual restart) affecting Cumulus close to rollover time (should not restart within one "standard interval" plus one minute, before or after)
| Use [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|editor in MX admin interface]], amend one line, and delete other
| See [[#Using the Cumulus 1 editing feature]]. Looking at the duplicate lines, correct one line, using information from it or from the duplicates, then click '''Delete''' to remove the uncorrected line(s)
|-
| Lines not in ascending date order
| Example possibilities:
# Bug in some Cumulus 1.9.2 builds sorted lines into wrong order if PC used USA date format (m-d-y), corrected from 1.9.3 onwards
# Restarting Cumulus after a crash, either manually using "Rewind" approach or Cumulus is confused by a corrupted file
| Manual editing outside MX in a text editor, if particular dates appear twice, see [[#Dates restart/repeat]]
| Manual editing outside Cumulus in a text editor, if particular dates appear twice, see [[#Dates restart/repeat]]
|-
| Some lines with fewer fields than others
| As explained at [[Dayfile.txt#List_of_Fields]], as Cumulus has developed, more fields have been added
| Use [[Calculate_Missing_Values#CreateMissing.exe|CreateMissing.exe]]
| Use workaround described at [[#Legacy Workaround]]
|-
| Some dates (lines) missing
| Examples:
* Inserting data before you start using Cumulus, see [[#Importing data not recorded by Cumulus]]
* Cumulus fails during rollover, so a line is not stored
| Use [[Calculate_Missing_Values#CreateMissing.exe|CreateMissing.exe]]
| Use [[#'''Create Missing''' on legacy dayfile editor]]
|}
 
 
==Missing or Corrupted past ''dayfile.txt'' lines in any Cumulus software==
 
If you accidentally corrupt a line in the [[dayfile.txt|daily summary log]], then view the resulting file:
* MX: Use [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|MX's admin interface]]
* C1: Use [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|original Cumulus software]]
 
Note that both flavours have editors that can be used to check the file contents, even if you don't need to edit any line.
 
If you have one, or more, dates missing in your dayfile.txt file, then the next question is, '''has the line been deleted by accident?'''
 
If a line for a particular date was present before, but is now corrupted or missing:
#See if you have a back-up of dayfile.txt with the line present and correct
#*If it is a recent date, then Cumulus makes a backup of dayfile.txt every time it is restarted and after every end-of-day rollover
#*If it is an older date, then maybe you took a back-up onto a separate drive or separate device
#If you have a suitable backup available, take a copy of that file
#Append onto the backup, any dates after when that copy ends, taking the extra lines from the current dayfile.txt
#Rename the current dayfile.txt to say dayfile.old
#Rename the copy you have edited to dayfile.txt and place into '''[[Data folder|data]]''' sub-folder
#Cumulus will now use the file with all days correct
 
 
''One lesson here, is to try to remember (once a week), to check your dayfile.txt log file is okay, because Cumulus retains back-ups for only the last 7 days''
 
'''Another lesson here, is to periodically take a backup, stored away from your Cumulus running environment in case you ever corrupt an old line'''
 
==Dates restart/repeat==
 
If you have a Cumulus crash (either because the connection between the weather station and Cumulus fails; or because there is a untrapped error in the Cumulus code; or because there is a power issue), then when it restarts it is possible that Cumulus will think it is doing catch-up from an earlier day, and the ''dayfile.txt'' may end up with consecutive lines being in date ascending order until the problem, then jumping back to an earlier date, before continuing in date asceneding order (but repeating one, or more, dates already in file).
 
If Cumulus was working correctly before the problem, then the lines stored before the problem should be okay, just delete the lines that repeat earlier dates, so the file ends up being date ascending order with no dupicates. Similarly, if Cumulus was working correctly after the problem, but there was an issue before the restart, then keep the lines that repeat the dates, but delete the earlier lines with same dates, so the file ends up being date ascending order with no dupicates.
 
If there are two lines with the date when the problem occured, then it is likely you will manually have to edit the two lines into one line.  Any field with a time-stamp before the problem will be kept unless it is obvious that extreme was correctly broken (i.e. not rogue restart value) after the problem.
 
If the dates restart in your daily summary log file, because you manually stopped Cumulus close to a rollover time, or you corrupted a file perhaps by regressing to an older release and back; then it is likely you will need to merge the two lines with same date, deciding for each value field which is more likely to be right, and matching it with correct time-stamp. A rainfall (or wind run) total might require summing totals in the two individual lines, or discarding a rouge value and accepting the other, your judgement!


==Validation by in-built editors==
==Validation by in-built editors==
Line 31: Line 129:
The legacy editor does also validate individual fields for appropriate content (integer, real number, time-stamp).  
The legacy editor does also validate individual fields for appropriate content (integer, real number, time-stamp).  


Unfortunately, the editor provided with MX does not validate any fields.  Thus you can use the wrong separator (between date elements, between hour and minute for time-stamps, between integer and decimal parts in any real number) in the MX editor and the incorrect line will be accepted.   
Unfortunately, '''the editor provided with MX does not validate any fields'''.  Thus you can use the wrong separator (between date elements, between hour and minute for time-stamps, between integer and decimal parts in any real number) in the MX editor and the incorrect line will be accepted.   


==Correcting date separator errors==
==Correcting date separator errors==


The legacy software accepts any character (except space) as the separator between the day of month, the month, and the year, elements of the date.  Therefore Cumulus 1 does not care if that separator is sometimes "/", sometimes "-", and/or sometimes ".".
=== Date Separator in MX===
 
MX is fussy, the same separator must be used in every date in the file. An error will be reported in diagnostics, and the historic graphs will stop working if the separator changes from one line to another.
MX is fussy, the same separator must be used in every date in the file. An error will be reported in diagnostics, and the historic graphs will stop working if the separator changes from one line to another.
# So look at the last line in the file, that is the character MX expects
# So look at the last line in the file, that is the character MX expects
Line 47: Line 144:
See [https://cumulus.hosiene.co.uk/viewtopic.php?f=40&t=18340 this forum thread] for a fuller discussion.
See [https://cumulus.hosiene.co.uk/viewtopic.php?f=40&t=18340 this forum thread] for a fuller discussion.


===Date Separator in Legacy Software===
The legacy software accepts any character (except space) as the separator between the day of month, the month, and the year, elements of the date.  Therefore Cumulus 1 does not care if that separator is sometimes "/", sometimes "-", and/or sometimes ".".


Historic note: In Cumulus 1 builds, any symbol (including ;, :, /, &, -, and .) character (excluding what is defined as list separator) within the first field of each line was treated as part of separator characters when parsing for date. For builds 3000 to build 3049, MX used fixed offsets to find day, month, year. From Build 3050 MX uses the separators defined by the OS to split the values (so allowing for multi-character separators), but it does mean it will be sensitive to the separator value changing.
Historic note: In Cumulus 1 builds, any symbol (including ;, :, /, &, -, and .) character (excluding what is defined as list separator) within the first field of each line was treated as part of separator characters when parsing for date. For builds 3000 to build 3049, MX used fixed offsets to find day, month, year. From Build 3050 MX uses the separators defined by the OS to split the values (so allowing for multi-character separators), but it does mean it will be sensitive to the separator value changing.
Line 52: Line 151:
= Importing data not recorded by Cumulus =
= Importing data not recorded by Cumulus =


You might have been using your weather station with some other weather software before you installed Cumulus.  If you can get weather data in the format of daily summaries (and the rollover times match), you can import that data into the Cumulus dayfile.txt file using a script or spreadsheet package.  All you have to ensure is that you can arrange the output to be in lines with fields in sequence shown in [[Dayfile.txt#List_of_Fields]].  There is more guidance later on this page about the rules you must obey for this file.
You might have been using your weather station with some other weather software before you installed Cumulus.  If you can get weather data in the format of daily summaries (and the rollover times match), you can import that data into the Cumulus '''dayfile.txt''' file using a script or spreadsheet package.  All you have to ensure is that you can arrange the output to be in lines with fields in sequence shown in [[Dayfile.txt#List_of_Fields]].  There is more guidance later on this page about the rules you must obey for this file.


If you have imported the data from the other weather software into the [[Standard_log_files]] format, then in the Cumulus 1 editor, ''Create missing'' can insert the new rows for those days previously missing in dayfile.txt, as explained below.  
If you have imported the data from the other weather software into the [[Standard_log_files]] format, then in the Cumulus 1 editor, ''Create missing'' can insert the new rows for those days previously missing in dayfile.txt, as explained below.  
= Using CumulusMX.exe editing functionality =
The editor in MX can be found in the administrative interface by selecting [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|Data Logs menu and '''Dayfile''' page]]. The '''Dayfile viewer/editor'' will display some lines at a time (e.g. 10 at a time or more). When the page is first loaded, the oldest lines will be read from the file by the Cumulus MX engine, and via an '''application programming interface''' (api) transferred to the web page where the lines are displayed using some software called ''datatables''. That software generates a navigation section where you can navigate to First, Previous, Next, and Last, with (for a longer file) up to 6 page numbers (each containing up to the selected number of lines) that you can select directly.
Basically, this is a text editor, but you have to amend lines using a pop-up, a proposal to use in-line editing never made it into a formal release. Another pop-up is used to confirm dleting a line.  The editor does not let you edit the date field.  Each line is identified by a line number.
This editor does not provide any way of inserting new lines into dayfile.txt (so you cannot correct an error when MX end of day failed and a line was not created), nor is there any way of changing the dates used by lines in the file (a very common problem reported by Cumulus users is that MX is giving them problems because not all lines in this file use the same format for the date, but this editor cannot resolve that).
There is a '''Refresh''' button that sends a '''application programming interface''' instruction asking the MX engine to resend the lines on the currently selected page via the api.
When you select a line in the provided editor, both  '''Edit''' and '''Delete'''  buttons are enabled.
Pick '''Edit''', click that, and an editing dialog pops up (MX uses '''altEditor ''' software for this). The pop up window does not let you change the line number nor the date, but all other fields show their current contents and you can overtype as necessary. Scroll down to see 2 buttons (how they are labelled depends on which version you are using), the left hand button ignores any edits you have made (it is labelled 'Close' or "Cancel" and simply does same effect as clicking the "x" in the top right corner), it prevents the api sending any replace message back to the MX engine. The right hand button saves your changes (even if it is labelled 'Edit' rather than "Save" in the version you are using) by using the api to send the replacement array back to the MX engine where it will replace the relevant line number before writing back to the log file.
There is no validation in the MX editor that was set up relatively quickly in version 3.4.5 as the first of 3 log file editors to plug a gap in MX functionality in earlier versions:
*some fields can only accept integers, other expect decimals,
* and some fields can accept negatives, others don't accept signed numbers
*some fields have a minimum and/or maximum acceptable value
As all lines are passed back via an application programme interface to the MX engine, there is no validation there either, the new line replaces the old one when the whole file is recreated. It is likely that the next time MX attempts to read the dayfile.txt it will find any error.  There was a third-party proposal to add simple validation into a replacement editor, that would hav
Pick '''Delete''', click that, and a simple dialog pops up (MX uses '''altEditor ''' software for this) showing all the fields in the selected line and asking you to confirm that you want to delete it. Again, the labelling on the buttons varies depending on which version you are running, one confirms the deletion (which sends the array back to the MX engine with an instruction that line number is to be deleted.  Despite the MX engine getting a copy of the fields that are to be deleted, it does not validate the fields, just the line number. The button labelled 'Close' or 'Cancel' does the same effect as clicking the "x" in the top right corner, it prevents the api sending any deletion message back to the MX engine.
==Cautions if using an obsolete MX release==
* '''Cumulus MX beta version 3.0.0 (checked at build 3043) does not provide an editor'''
* '''Cumulus MX Version 3.4.5 - Build 3069 onwards provides an editor'''
* From MX release 3.9.2 - b3097 historic charts are added to admin interface
** If you use the in-built editor to update dayfile.txt, your change will be reflected in the historic graphs
** If you edit dayfile.txt outside Cumulus MX, historic graphs will not be updated until Cumulus MX is restarted
If lines in your [[dayfile.txt|daily summary log]] were created by  a variety of Cumulus 1 releases (so some have less than 46 fields), you need to use Cumulus MX version 3.7.0, or later, to be sure that the provided editor will cope. The code was actually amended to be able to read lines with fewer fields at version 3.6.0, when 4 fields were added for feels like, so the total number of fields became 50.
It is recommended that nobody uses any 3.6.x version (see [[Updating_MX_to_new_version#If_using_a_3.5.x_release]]. (For historical interest only, Emergency Version 3.6.12 changed the number of fields to 54 in error)
From formal release 3.7.0, MX's dayfile.txt reverted to 52 fields. The extras are Canadian Humidity Index (Humidex). At the time of typing this, it has been said this file structure will not be changed again.
Whatever MX release you use, any line that is edited (even if it originally had fewer or more fields) will be saved with the same number of fields that release uses when it creates a line from the contents of today.ini.
= Using CreateMissing.exe editing functionality=
'''You must be running MX in order to use this utility''', as some information is shared between CumulusMX.exe and CreateMissing.exe.
Please see [[Calculate_Missing_Values#CreateMissing.exe|CreateMissing.exe]] and/or https://github.com/cumulusmx/CreateMissing/blob/master/README.md.
Information there is not repeated here.
Basically, this utility will not replace any data that is already in the daily summary log file. It will create new lines if data for those missing lines can be derived from data already in [[Standard_log_files]]. If an individual line in '''dayfile.txt''' is shorter than the number of fields expected in the current MX release, then '''CreateMissing.exe''' will calculate daily values for the missing fields based on data already in [[Standard_log_files]].




Line 67: Line 218:
*you can use '''insert''' key to add one or more missing rows (complete days) manually typing in values for all fields,
*you can use '''insert''' key to add one or more missing rows (complete days) manually typing in values for all fields,
*use '''delete''' key to remove an entire day (e.g. if you get a 'duplicate' error message) after ensuring all fields are correct in the line that will remain,
*use '''delete''' key to remove an entire day (e.g. if you get a 'duplicate' error message) after ensuring all fields are correct in the line that will remain,
*use '''Create missing''' button to insert missing rows (complete days) by reading from [[Standard_log_files]] and automatically calculating the best approximations for each field for those missing days.
*use '''Create missing''' button to insert missing rows (complete days) by reading from [[Standard_log_files]] and automatically calculating the best approximations for each field for those missing days (see next section)


== Create Missing ==
== '''Create Missing''' on legacy dayfile editor ==


If a date is missing from dayfile.txt, and not available in a back-up, then it is possible to create a missing line (with approximations for the derived extreme values) in Cumulus 1.  That functionality is known as "Create Missing" and is found [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|within the edit dayfile.txt screen]].
If a date is missing from dayfile.txt, and not available in a back-up, then it is possible to create a missing line (with approximations for the derived extreme values) in Cumulus 1.  That functionality is known as "Create Missing" and is found [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|within the edit dayfile.txt screen]].
Line 87: Line 238:
If a particular day does not exist as a row on the daily summary log, then 'create missing' can search the observations in the relevant monthly log, and calculate approximate highs, lows and totals to insert as an extra row in the daily summary log. These are approximate because the actual highs and lows for that day are quite likely to have occurred at moments in-between those that were logged.  
If a particular day does not exist as a row on the daily summary log, then 'create missing' can search the observations in the relevant monthly log, and calculate approximate highs, lows and totals to insert as an extra row in the daily summary log. These are approximate because the actual highs and lows for that day are quite likely to have occurred at moments in-between those that were logged.  


For ''Create missing'' a list of inserted records is produced in [[dayfileeditlog.txt]].  
For the legacy ''Create missing'' a list of inserted records is produced in [[dayfileeditlog.txt]].  


If just some fields are wrong in a particular row (meteorological day) on day file, then there is a [[Standard_log_files#Using_Monthly_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | work around]] as at all current versions (up to 1.9.4) you can only use 'Create missing' to read from the [[Standard_log_files]] if the whole day (a line starting with that date) is missing in ''dayfile.txt''.  Although Cumulus does not recognise the concept of a sensor not being available, it will write solar information even if you don't have a solar sensor; it does have to cope with reading a monthly log file that might have fewer derivatives than it wants (when using Create Missing) and therefore it may not know what to write into dayfile.txt as the calculated value. Cumulus 1 can't write a null value, so it writes zero for values, and "00:00" for time stamps. If you are using a 9am or 10am rollover time, be aware that create missing in Cumulus 1 always inserts 00:00 for null time-stamps, but in normal running Cumulus uses the rollover time for null time-stamps.
If just some fields are wrong in a particular row (meteorological day) on day file, then there is a [[Standard_log_files#Using_Monthly_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | work around]] as at all current versions (up to 1.9.4) you can only use 'Create missing' to read from the [[Standard_log_files]] if the whole day (a line starting with that date) is missing in ''dayfile.txt''.  Although Cumulus does not recognise the concept of a sensor not being available, it will write solar information even if you don't have a solar sensor; it does have to cope with reading a monthly log file that might have fewer derivatives than it wants (when using Create Missing) and therefore it may not know what to write into dayfile.txt as the calculated value. Cumulus 1 can't write a null value, so it writes zero for values, and "00:00" for time stamps. If you are using a 9am or 10am rollover time, be aware that create missing in Cumulus 1 always inserts 00:00 for null time-stamps, but in normal running Cumulus uses the rollover time for null time-stamps.


== Dealing with errors identified by the software ==
== Dealing with errors identified by the legacy software ==


If there is an error in ''dayfile.txt'', then it is most likely to be found when you are viewing its data on one of the screens for editing the monthly, annual or all-time extremes.  Cumulus 1 will illuminate its ''Error'' light if it finds an error in such cases and tell you the line/row number of the first found error, together with some details of the error it found. For example, if a row is blank, a row is duplicated, a field is corrupted, a field does not have an acceptable value, or a field is missing so subsequent fields are to the left of where they should be.
If there is an error in ''dayfile.txt'', then it is most likely to be found when you are viewing its data on one of the screens for editing the monthly, annual or all-time extremes.  Cumulus 1 will illuminate its ''Error'' light if it finds an error in such cases and tell you the line/row number of the first found error, together with some details of the error it found. For example, if a row is blank, a row is duplicated, a field is corrupted, a field does not have an acceptable value, or a field is missing so subsequent fields are to the left of where they should be.
Line 98: Line 249:




==Missing or Corrupted dayfile.txt lines==
==Correcting individual fields in multiple dayfile.txt lines==


If you accidentally corrupt a line in the [[dayfile.txt|daily summary log]], then both the [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|original Cumulus software]] and [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|MX's admin interface]] have editors for viewing and modifying content.
Often people want to correct an individual field, in all lines within a particular period.


If you have one, or more, dates missing in your dayfile.txt file, then the next question is, '''has the line been deleted by accident?'''
The most common example is because you have suddenly discovered your daily pressure high (and low) are wrong by a certain offset, because you used the wrong setting for converting absolute pressure to the sea level equivalent you want to store in the file.  
 
If a line for a particular date was present before, but is now corrupted or missing:
#See if you have a back-up of dayfile.txt with the line present and correct
#*If it is a recent date, then Cumulus makes a backup of dayfile.txt every time it is restarted and after every end-of-day rollover
#*If it is an older date, then maybe you took a back-up onto a separate drive or separate device
#If you have a suitable backup available, take a copy of that file
#Append onto the copy, any dates after when that copy ends, taking the extra lines from the current dayfile.txt
#Rename the current dayfile.txt to say dayfile.old
#Rename the copy you have edited to dayfile.txt and place into '''[[Data folder|data]]''' sub-folder
#Cumulus will now use the file with all days correct
 
 
''One lesson here, is to try to remember (once a week), to check your dayfile.txt log file is okay, because Cumulus retains back-ups for only the last 7 days''
 
'''Another lesson here, is to periodically take a backup, stored away from your Cumulus running environment in case you ever corrupt an old line'''
 
 
==Correcting individual fields in existing dayfile.txt lines==
 
The "Create Missing" functionality, described in earlier sub-section, is designed for when an entire line is missing for a prior date.  Often people want to correct an individual field, in all lines within a particular period.  This might be because you have suddenly discovered your daily pressure high and low are wrong by a certain offset, because you used the wrong setting for converting absolute pressure to the sea level equivalent you want to store in the file.  


Neither Cumulus 1 nor MX provide any functionality to make such multi-line editing easy.  
Neither Cumulus 1 nor MX provide any functionality to make such multi-line editing easy.  


However, this can be resolved by opening dayfile.txt in a spreadsheet (carefully ensuring each column is treated as either text (date and time fields) or number (the value fields) and that your field separator is set in the filter conditions to be a column separator. Spreadsheets have a "Paste Special" function that allows you to add/subtract a constant to every cell in a column.  
However, this can be resolved by opening dayfile.txt in a spreadsheet (carefully ensuring each column is treated as either '''text''' (date and time fields) or number (the value fields) and that '''your field separator''' is set in the '''filter conditions''' to be a column separator for converting to spreadsheet. Spreadsheets have a "Paste Special" function that allows you to add/subtract a constant to every cell in a column.  


==Correcting individual fields in multiple lines of Daily summary Log in Cumulus 1==
===Correcting individual fields in multiple lines of Daily summary Log in Cumulus 1===


In the '''Edit''' menu, select '''Dayfile.txt''' screen, for a text editor where you can overwrite entries, delete entire lines, and insert new lines (and then manually type in values for every field). You can scroll left and right between fields, with a header row identifying which field is which, and you can scroll up and down through all the lines.  If you know what you want to change (such as replacing a rogue figure), this is a very easy to use editor.
In the '''Edit''' menu, select '''Dayfile.txt''' screen, for a text editor where you can overwrite entries, delete entire lines, and insert new lines (and then manually type in values for every field). You can scroll left and right between fields, with a header row identifying which field is which, and you can scroll up and down through all the lines.  If you know what you want to change (such as replacing a rogue figure), this is a very easy to use editor.


For the daily summary log, the original Cumulus software includes functionality to '''Create Missing''' in its daily summary log editor, see [[Amending_dayfile#Create_Missing]].  This functionality creates an entire missing log line and will insert highs and lows whether they are extremes of source values, or extremes of derived values.  
For the daily summary log, the original Cumulus software includes functionality to '''Create Missing''' in its daily summary log editor, see [[# '''Create Missing''' on legacy dayfile editor]].  This functionality creates '''an entire missing log line'' and will insert highs and lows whether they are extremes of source values, or extremes of derived values. It will only add those fields (to any new line) that are maximum or minimum (or sum, or equal to) a field that does exist in the [[Standard_log_files]] lines that belong to the same meteorological day.


== Workaround==
==Legacy Workaround==


This functionality is not designed to work out individual missing extreme figures, only to insert complete missing lines. However, we can workaround that constraint:
The legacy functionality is not designed to insert (or correct) individual missing extreme figures for daily summary lines that already have some fields in them, only to insert complete missing lines. However, we can workaround that constraint:


'''WORKAROUND FOR DAYFILE.TXT if required dates are present in both the standard log and dayfile.txt, but not all fields for that date exist in dayfile.txt'''
'''WORKAROUND FOR DAYFILE.TXT if required dates are present in both the standard log and dayfile.txt, but not all fields for that date exist in dayfile.txt'''
Line 169: Line 300:
*Cumulus 1 does not have the functionality to retrospectively calculate derived values when it is reading source values in the standard data log line.
*Cumulus 1 does not have the functionality to retrospectively calculate derived values when it is reading source values in the standard data log line.


= [[File:Badge vMx.png]]Using the Cumulus MX editing functionality=
*'''Cumulus MX beta version 3.0.0 (checked at build 3043) does not provide an editor'''
*'''Cumulus MX Version 3.4.5 - Build 3069 onwards provides an editor'''
*From MX release 3.9.2 - b3097 historic charts are added to admin interface
**If you use the in-built editor to update dayfile.txt, your change will be reflected in the historic graphs
**If you edit dayfile.txt outside Cumulus MX, historic graphs will not be updated until Cumulus MX is restarted
There is a dayfile.txt editor within the admin interface to edit this log file.  In the '''Edit''' menu, choose the '''Dayfile''' option.
If lines in your [[dayfile.txt|daily summary log]] were created by  a variety of Cumulus 1 releases (so some have less than 46 fields), you need to use Cumulus MX version 3.7.0, or later, to be sure that the provided editor will cope. The code was actually amended to be able to read lines with fewer fields at version 3.6.0, when 4 fields were added for feels like, so the total number of fields became 50.
It is recommended that nobody uses any 3.6.x version (see [[Updating_MX_to_new_version#If_using_a_3.5.x_release]]. (For historical interest only, Emergency Version 3.6.12 changed the number of fields to 54 in error)
From formal release 3.7.0, MX's dayfile.txt reverted to 52 fields. The extras are Canadian Humidity Index (Humidex). At the time of typing this, it has been said this file structure will not be changed again.
Whatever MX release you use, any line that is edited (even if it originally had fewer or more fields) will be saved with the same number of fields that release uses when it creates a line from the contents of today.ini.
=== Editing  inside MX ===
The editor in MX can be found in the administrative interface by selecting [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|Data Logs menu and '''Dayfile''' page]]. The '''Dayfile viewer/editor'' will display up to 10 lines at a time. When the page is first loaded, the oldest lines will be read from the file by the Cumulus MX engine, and via an '''application programming interface''' (api) transferred to the web page where the lines are displayed using some software called ''datatables''. That software generates a navigation section where you can navigate to First, Previous, Next, and Last, with (for a longer file) up to 6 page numbers (each containing up to 10 lines) that you can select directly.
The editor does not provide any way of inserting new lines into dayfile.txt (so you cannot correct an error when MX end of day failed and a line was not created), nor is there any way of changing the dates used by lines in the file (a very common problem reported by Cumulus users is that MX is giving them problems because not all lines in this file use the same format for the date, but this editor cannot resolve that).
There is a '''Refresh''' button that sends a '''application programming interface''' instruction asking the MX engine to resend the lines on the currently selected page via the api.
When you select a line in the provided editor, both  '''Edit''' and '''Delete'''  buttons are enabled.
Pick '''Edit''', click that, and an editing dialog pops up (MX uses '''altEditor ''' software for this). The pop up window does not let you change the line number nor the date, but all other fields show their current contents and you can overtype as necessary. Scroll down to see 2 buttons (how they are labelled depends on which version you are using), the left hand button ignores any edits you have made (it is labelled 'Close' or "Cancel" and simply does same effect as clicking the "x" in the top right corner), it prevents the api sending any replace message back to the MX engine. The right hand button saves your changes (even if it is labelled 'Edit' rather than "Save" in the version you are using) by using the api to send the replacement array back to the MX engine where it will replace the relevant line number before writing back to the log file.
There is no validation in the MX editor that was set up relatively quickly in version 3.4.5 as the first of 3 log file editors to plug a gap in MX functionality in earlier versions:
*some fields can only accept integers, other expect decimals,
* and some fields can accept negatives, others don't accept signed numbers
*some fields have a minimum and/or maximum acceptable value
As all lines are passed back via an application programme interface to the MX engine, there is no validation there either, the new line replaces the old one when the whole file is recreated. It is likely that the next time MX attempts to read the dayfile.txt it will find any error.  There was a third-party proposal to add simple validation into a replacement editor, that would hav
Pick '''Delete''', click that, and a simple dialog pops up (MX uses '''altEditor ''' software for this) showing all the fields in the selected line and asking you to confirm that you want to delete it. Again, the labelling on the buttons varies depending on which version you are running, one confirms the deletion (which sends the array back to the MX engine with an instruction that line number is to be deleted.  Despite the MX engine getting a copy of the fields that are to be deleted, it does not validate the fields, just the line number. The button labelled 'Close' or 'Cancel' does the same effect as clicking the "x" in the top right corner, it prevents the api sending any deletion message back to the MX engine.


= Important Rules when editing dayfile.txt =
= Important Rules when editing dayfile.txt =

Revision as of 18:39, 18 June 2021

Cumulus uses a daily summary log file, the fields in that file are listed at Dayfile.txt#List_of_Fields. The information about amending the file that was originally on the same page has been moved to this page.

This page was created with text for the (legacy) Cumulus 1 software, but it has been amended to cover MX as well. Cumulus Version 1 SpecificCumulus Version MX Specific

How Cumulus uses the daily summary log

While Cumulus is reading your weather station, all daily extremes, counts, and totals, Cumulus needs are stored in today.ini.

Reading the daily summary log file

For MX release 3.9.2 - build 3097 onwards, when the software first starts, the whole of dayfile.txt is read, the contents are used to drive the Highcharts_-_Historic functionality.

The file is also read, if you are using the editor provided in Cumulus.

Writing a new line to the daily summary log file

At the end of each meteorological day, Cumulus uses information held in today.ini to write a new line into dayfile.txt, before it resets today.ini ready for the new day.

Chill Hours Daily Increment for storing in dayfile.txt is calculated by substracting the cumulative count in yesterday.ini from the cumulative count in today.ini.

If rollover, is not at midnight, sunshine hours (these run from midnight to midnight) for the dayfile.txt are read from yesterday.ini. For rollover at midnight, sunshine hours from today.ini (before reset) is read.


Problems that occur in the daily summary log and how to safely rectify them

As discussed in Correcting_Extremes, it is possible for rogue values to be read from a weather station, and propagate into various log files. An error in many of those files, corrupts a particular extreme record (or more than one), but generally does not stop Cumulus working.

Since MX now reads the entire daily summary log when you start it up, an error in dayfile.txt can cause a more significant problem, most likely for historic charts, but it can affect other functionality!


Summary for most common problems

Problem Possible cause Rectification for MX releases Rectification for legacy versions
One, or more, past line(s) accidentally deleted or corrupted A mistake during a manual view/edit to the file, or a mistake while using the built-in editor See #Missing or Corrupted past dayfile.txt lines in any Cumulus software See #Missing or Corrupted past dayfile.txt lines in any Cumulus software
Inconsistencies in date formats (what comes between day of month, month, and year) Examples:
  1. File was previously used with Cumulus 1 (that did not care what symbol came between parts of date)
  2. Moving from one device to another (and not ensuring same locale on both devices)
  3. Editing using an external app (like Excel) and treating date cells as date type, should be text type
Cannot use editor in MX admin interface, for bulk line edits see #Correcting date separator errors Type over individual dates in built-in editor, for bulk line edits see #Correcting date separator errors
Inconsistencies between decimal comma and decimal points Moving from one device to another (and not ensuring same locale on both devices) Correct "locale" and/or use an external editor that offers "Replace all" (see #Validation by in-built editors) Use Control Panel to correct region. Correct "locale" and/or use an external editor that offers "Replace all"
Inconsistencies in list separator (what comes between fields) Moving from one device to another (and not ensuring same locale on both devices) Correct "locale" and/or use an external editor that offers "Replace all" Use Control Panel to correct region. See #Using the Cumulus 1 editing feature to check file is now consistent.
Duplication of dates between lines (either consecutive lines, or non-adjacient lines) Electrical fault (or manual restart) affecting Cumulus close to rollover time (should not restart within one "standard interval" plus one minute, before or after) Use editor in MX admin interface, amend one line, and delete other See #Using the Cumulus 1 editing feature. Looking at the duplicate lines, correct one line, using information from it or from the duplicates, then click Delete to remove the uncorrected line(s)
Lines not in ascending date order Example possibilities:
  1. Bug in some Cumulus 1.9.2 builds sorted lines into wrong order if PC used USA date format (m-d-y), corrected from 1.9.3 onwards
  2. Restarting Cumulus after a crash, either manually using "Rewind" approach or Cumulus is confused by a corrupted file
Manual editing outside MX in a text editor, if particular dates appear twice, see #Dates restart/repeat Manual editing outside Cumulus in a text editor, if particular dates appear twice, see #Dates restart/repeat
Some lines with fewer fields than others As explained at Dayfile.txt#List_of_Fields, as Cumulus has developed, more fields have been added Use CreateMissing.exe Use workaround described at #Legacy Workaround
Some dates (lines) missing Examples: Use CreateMissing.exe Use #Create Missing on legacy dayfile editor


Missing or Corrupted past dayfile.txt lines in any Cumulus software

If you accidentally corrupt a line in the daily summary log, then view the resulting file:

Note that both flavours have editors that can be used to check the file contents, even if you don't need to edit any line.

If you have one, or more, dates missing in your dayfile.txt file, then the next question is, has the line been deleted by accident?

If a line for a particular date was present before, but is now corrupted or missing:

  1. See if you have a back-up of dayfile.txt with the line present and correct
    • If it is a recent date, then Cumulus makes a backup of dayfile.txt every time it is restarted and after every end-of-day rollover
    • If it is an older date, then maybe you took a back-up onto a separate drive or separate device
  2. If you have a suitable backup available, take a copy of that file
  3. Append onto the backup, any dates after when that copy ends, taking the extra lines from the current dayfile.txt
  4. Rename the current dayfile.txt to say dayfile.old
  5. Rename the copy you have edited to dayfile.txt and place into data sub-folder
  6. Cumulus will now use the file with all days correct


One lesson here, is to try to remember (once a week), to check your dayfile.txt log file is okay, because Cumulus retains back-ups for only the last 7 days

Another lesson here, is to periodically take a backup, stored away from your Cumulus running environment in case you ever corrupt an old line

Dates restart/repeat

If you have a Cumulus crash (either because the connection between the weather station and Cumulus fails; or because there is a untrapped error in the Cumulus code; or because there is a power issue), then when it restarts it is possible that Cumulus will think it is doing catch-up from an earlier day, and the dayfile.txt may end up with consecutive lines being in date ascending order until the problem, then jumping back to an earlier date, before continuing in date asceneding order (but repeating one, or more, dates already in file).

If Cumulus was working correctly before the problem, then the lines stored before the problem should be okay, just delete the lines that repeat earlier dates, so the file ends up being date ascending order with no dupicates. Similarly, if Cumulus was working correctly after the problem, but there was an issue before the restart, then keep the lines that repeat the dates, but delete the earlier lines with same dates, so the file ends up being date ascending order with no dupicates.

If there are two lines with the date when the problem occured, then it is likely you will manually have to edit the two lines into one line. Any field with a time-stamp before the problem will be kept unless it is obvious that extreme was correctly broken (i.e. not rogue restart value) after the problem.

If the dates restart in your daily summary log file, because you manually stopped Cumulus close to a rollover time, or you corrupted a file perhaps by regressing to an older release and back; then it is likely you will need to merge the two lines with same date, deciding for each value field which is more likely to be right, and matching it with correct time-stamp. A rainfall (or wind run) total might require summing totals in the two individual lines, or discarding a rouge value and accepting the other, your judgement!

Validation by in-built editors

Both the legacy editor and the MX editor will ensure that the correct number of fields is stored (as defined at the release where you do editing) (one common error in an external editor is to accidentally add/delete a field).

The legacy editor does also validate individual fields for appropriate content (integer, real number, time-stamp).

Unfortunately, the editor provided with MX does not validate any fields. Thus you can use the wrong separator (between date elements, between hour and minute for time-stamps, between integer and decimal parts in any real number) in the MX editor and the incorrect line will be accepted.

Correcting date separator errors

Date Separator in MX

MX is fussy, the same separator must be used in every date in the file. An error will be reported in diagnostics, and the historic graphs will stop working if the separator changes from one line to another.

  1. So look at the last line in the file, that is the character MX expects
  2. If any earlier line, or lines, uses a different character, then that line, or lines, must be edited, it is best to use an editor designed for programmers
    • If the last line has hyphen "-", but some earlier lines use slash "/", it is simple to do an edit
      • Use "Replace All" option in your chosen editor, to find "/", and replace with "-". This works because "/" does not appear anywhere else in the file
    • If the last line has "/", but some earlier lines have "." or "-" as separator, then the correction is more complex, because you don't want to edit any minus signs or decimal points in any other fields
      • See if your editor accepts "Regular Expressions", then you can look for two matching separators in same field, see this forum post.

See this forum thread for a fuller discussion.

Date Separator in Legacy Software

The legacy software accepts any character (except space) as the separator between the day of month, the month, and the year, elements of the date. Therefore Cumulus 1 does not care if that separator is sometimes "/", sometimes "-", and/or sometimes ".".

Historic note: In Cumulus 1 builds, any symbol (including ;, :, /, &, -, and .) character (excluding what is defined as list separator) within the first field of each line was treated as part of separator characters when parsing for date. For builds 3000 to build 3049, MX used fixed offsets to find day, month, year. From Build 3050 MX uses the separators defined by the OS to split the values (so allowing for multi-character separators), but it does mean it will be sensitive to the separator value changing.

Importing data not recorded by Cumulus

You might have been using your weather station with some other weather software before you installed Cumulus. If you can get weather data in the format of daily summaries (and the rollover times match), you can import that data into the Cumulus dayfile.txt file using a script or spreadsheet package. All you have to ensure is that you can arrange the output to be in lines with fields in sequence shown in Dayfile.txt#List_of_Fields. There is more guidance later on this page about the rules you must obey for this file.

If you have imported the data from the other weather software into the Standard_log_files format, then in the Cumulus 1 editor, Create missing can insert the new rows for those days previously missing in dayfile.txt, as explained below.

Using CumulusMX.exe editing functionality

The editor in MX can be found in the administrative interface by selecting Data Logs menu and Dayfile page. The Dayfile viewer/editor will display some lines at a time (e.g. 10 at a time or more). When the page is first loaded, the oldest lines will be read from the file by the Cumulus MX engine, and via an application programming interface' (api) transferred to the web page where the lines are displayed using some software called datatables. That software generates a navigation section where you can navigate to First, Previous, Next, and Last, with (for a longer file) up to 6 page numbers (each containing up to the selected number of lines) that you can select directly.

Basically, this is a text editor, but you have to amend lines using a pop-up, a proposal to use in-line editing never made it into a formal release. Another pop-up is used to confirm dleting a line. The editor does not let you edit the date field. Each line is identified by a line number.


This editor does not provide any way of inserting new lines into dayfile.txt (so you cannot correct an error when MX end of day failed and a line was not created), nor is there any way of changing the dates used by lines in the file (a very common problem reported by Cumulus users is that MX is giving them problems because not all lines in this file use the same format for the date, but this editor cannot resolve that).

There is a Refresh button that sends a application programming interface instruction asking the MX engine to resend the lines on the currently selected page via the api.

When you select a line in the provided editor, both Edit and Delete buttons are enabled.

Pick Edit, click that, and an editing dialog pops up (MX uses altEditor software for this). The pop up window does not let you change the line number nor the date, but all other fields show their current contents and you can overtype as necessary. Scroll down to see 2 buttons (how they are labelled depends on which version you are using), the left hand button ignores any edits you have made (it is labelled 'Close' or "Cancel" and simply does same effect as clicking the "x" in the top right corner), it prevents the api sending any replace message back to the MX engine. The right hand button saves your changes (even if it is labelled 'Edit' rather than "Save" in the version you are using) by using the api to send the replacement array back to the MX engine where it will replace the relevant line number before writing back to the log file.

There is no validation in the MX editor that was set up relatively quickly in version 3.4.5 as the first of 3 log file editors to plug a gap in MX functionality in earlier versions:

  • some fields can only accept integers, other expect decimals,
  • and some fields can accept negatives, others don't accept signed numbers
  • some fields have a minimum and/or maximum acceptable value

As all lines are passed back via an application programme interface to the MX engine, there is no validation there either, the new line replaces the old one when the whole file is recreated. It is likely that the next time MX attempts to read the dayfile.txt it will find any error. There was a third-party proposal to add simple validation into a replacement editor, that would hav

Pick Delete, click that, and a simple dialog pops up (MX uses altEditor software for this) showing all the fields in the selected line and asking you to confirm that you want to delete it. Again, the labelling on the buttons varies depending on which version you are running, one confirms the deletion (which sends the array back to the MX engine with an instruction that line number is to be deleted. Despite the MX engine getting a copy of the fields that are to be deleted, it does not validate the fields, just the line number. The button labelled 'Close' or 'Cancel' does the same effect as clicking the "x" in the top right corner, it prevents the api sending any deletion message back to the MX engine.


Cautions if using an obsolete MX release

  • Cumulus MX beta version 3.0.0 (checked at build 3043) does not provide an editor
  • Cumulus MX Version 3.4.5 - Build 3069 onwards provides an editor
  • From MX release 3.9.2 - b3097 historic charts are added to admin interface
    • If you use the in-built editor to update dayfile.txt, your change will be reflected in the historic graphs
    • If you edit dayfile.txt outside Cumulus MX, historic graphs will not be updated until Cumulus MX is restarted

If lines in your daily summary log were created by a variety of Cumulus 1 releases (so some have less than 46 fields), you need to use Cumulus MX version 3.7.0, or later, to be sure that the provided editor will cope. The code was actually amended to be able to read lines with fewer fields at version 3.6.0, when 4 fields were added for feels like, so the total number of fields became 50.

It is recommended that nobody uses any 3.6.x version (see Updating_MX_to_new_version#If_using_a_3.5.x_release. (For historical interest only, Emergency Version 3.6.12 changed the number of fields to 54 in error)

From formal release 3.7.0, MX's dayfile.txt reverted to 52 fields. The extras are Canadian Humidity Index (Humidex). At the time of typing this, it has been said this file structure will not be changed again.

Whatever MX release you use, any line that is edited (even if it originally had fewer or more fields) will be saved with the same number of fields that release uses when it creates a line from the contents of today.ini.

Using CreateMissing.exe editing functionality

You must be running MX in order to use this utility, as some information is shared between CumulusMX.exe and CreateMissing.exe.

Please see CreateMissing.exe and/or https://github.com/cumulusmx/CreateMissing/blob/master/README.md.

Information there is not repeated here.

Basically, this utility will not replace any data that is already in the daily summary log file. It will create new lines if data for those missing lines can be derived from data already in Standard_log_files. If an individual line in dayfile.txt is shorter than the number of fields expected in the current MX release, then CreateMissing.exe will calculate daily values for the missing fields based on data already in Standard_log_files.


Using the Cumulus 1 editing feature

Cumulus Version 1 SpecificThis section applies to Cumulus 1.x.y only. The last command in Edit menu is dayfile.txt.

This is how you view the dayfile.txt from within Cumulus. Click the Help button for detailed instructions. Cumulus Help is concise but comprehensive.

It is a text editor, and it works best when at full screen:

  • correct individual values by over-typing new values over those currently displayed,
  • you can use insert key to add one or more missing rows (complete days) manually typing in values for all fields,
  • use delete key to remove an entire day (e.g. if you get a 'duplicate' error message) after ensuring all fields are correct in the line that will remain,
  • use Create missing button to insert missing rows (complete days) by reading from Standard_log_files and automatically calculating the best approximations for each field for those missing days (see next section)

Create Missing on legacy dayfile editor

If a date is missing from dayfile.txt, and not available in a back-up, then it is possible to create a missing line (with approximations for the derived extreme values) in Cumulus 1. That functionality is known as "Create Missing" and is found within the edit dayfile.txt screen.

  1. Only use the editor when Cumulus is not doing catch-up or end-of-day, or any other action that could create an editing conflict.
  2. With the contents of the file loaded into the editing table, click the Create Missing button
  3. Cumulus 1 will now work through every line from start to end
  4. If 2 adjacent lines are not for consecutive dates, Cumulus will attempt to create missing lines
  5. It will read from the Standard log files all the data logged for each missing meteorological date (if 9am or 10am rollover is used, those read spans two calendar dates)
  6. It will derive totals, averages, highs, and lows, from the data it reads, for each missing date
    • Note that normally dayfile.txt lines are created from Today.ini which logs the daily totals, averages, highs, and lows, from every reading taken from the weather station
    • Depending on your weather station, Cumulus is able to read values every minute, and consequently update today.ini each minute if an extreme happens,
    • If Cumulus is set up to only log the readings every half an hour, create missing is only able to see 1/30th of the data,
    • Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence why getting missing lines from a backup is better)

The Cumulus 1 editor provides a "Create Missing" option where it will, for any dates for which a line does not exist, create a line if it can from reading the detailed log file to extract all values relevant to that day and do the necessary minimum/maximum/total/average calculation for each dayfile.txt field, storing the time from the relevant other log file in any time-stamp field in dayfile.txt.

If a particular day does not exist as a row on the daily summary log, then 'create missing' can search the observations in the relevant monthly log, and calculate approximate highs, lows and totals to insert as an extra row in the daily summary log. These are approximate because the actual highs and lows for that day are quite likely to have occurred at moments in-between those that were logged.

For the legacy Create missing a list of inserted records is produced in dayfileeditlog.txt.

If just some fields are wrong in a particular row (meteorological day) on day file, then there is a work around as at all current versions (up to 1.9.4) you can only use 'Create missing' to read from the Standard_log_files if the whole day (a line starting with that date) is missing in dayfile.txt. Although Cumulus does not recognise the concept of a sensor not being available, it will write solar information even if you don't have a solar sensor; it does have to cope with reading a monthly log file that might have fewer derivatives than it wants (when using Create Missing) and therefore it may not know what to write into dayfile.txt as the calculated value. Cumulus 1 can't write a null value, so it writes zero for values, and "00:00" for time stamps. If you are using a 9am or 10am rollover time, be aware that create missing in Cumulus 1 always inserts 00:00 for null time-stamps, but in normal running Cumulus uses the rollover time for null time-stamps.

Dealing with errors identified by the legacy software

If there is an error in dayfile.txt, then it is most likely to be found when you are viewing its data on one of the screens for editing the monthly, annual or all-time extremes. Cumulus 1 will illuminate its Error light if it finds an error in such cases and tell you the line/row number of the first found error, together with some details of the error it found. For example, if a row is blank, a row is duplicated, a field is corrupted, a field does not have an acceptable value, or a field is missing so subsequent fields are to the left of where they should be.

If you do have a 'duplicate' error, you need to decide which row to delete, and whether to copy any values from that row into the row you are keeping to ensure the correct extremes are retained.


Correcting individual fields in multiple dayfile.txt lines

Often people want to correct an individual field, in all lines within a particular period.

The most common example is because you have suddenly discovered your daily pressure high (and low) are wrong by a certain offset, because you used the wrong setting for converting absolute pressure to the sea level equivalent you want to store in the file.

Neither Cumulus 1 nor MX provide any functionality to make such multi-line editing easy.

However, this can be resolved by opening dayfile.txt in a spreadsheet (carefully ensuring each column is treated as either text (date and time fields) or number (the value fields) and that your field separator is set in the filter conditions to be a column separator for converting to spreadsheet. Spreadsheets have a "Paste Special" function that allows you to add/subtract a constant to every cell in a column.

Correcting individual fields in multiple lines of Daily summary Log in Cumulus 1

In the Edit menu, select Dayfile.txt screen, for a text editor where you can overwrite entries, delete entire lines, and insert new lines (and then manually type in values for every field). You can scroll left and right between fields, with a header row identifying which field is which, and you can scroll up and down through all the lines. If you know what you want to change (such as replacing a rogue figure), this is a very easy to use editor.

For the daily summary log, the original Cumulus software includes functionality to Create Missing' in its daily summary log editor, see # Create Missing on legacy dayfile editor. This functionality creates an entire missing log line and will insert highs and lows whether they are extremes of source values, or extremes of derived values. It will only add those fields (to any new line) that are maximum or minimum (or sum, or equal to) a field that does exist in the Standard_log_files lines that belong to the same meteorological day.

Legacy Workaround

The legacy functionality is not designed to insert (or correct) individual missing extreme figures for daily summary lines that already have some fields in them, only to insert complete missing lines. However, we can workaround that constraint:

WORKAROUND FOR DAYFILE.TXT if required dates are present in both the standard log and dayfile.txt, but not all fields for that date exist in dayfile.txt

There are two common reasons for needing to use this, Cumulus 1 only, workaround:

  1. To add extra derived fields now calculated by Cumulus, to past lines created by an earlier version of Cumulus that did not calculate those derived fields at the time
  2. If you are importing into Cumulus records from other software, and that other software does not provide derived extremes (e.g. dew point, apparent temperature, heat index, rain rate, wind run) so your insert in the daily summary log file has only populated a sub-set of the fields.

The steps you need to take, to implement this Cumulus 1 only workaround:

  1. Select a time to do this work that is at least 10 minutes, preferably a whole hour, after Cumulus has performed a daily rollover, and ensure you finish this work long before the next rollover time. This will ensure your work cannot conflict with when Cumulus 1 is updating dayfile.txt.
  2. Take a copy of dayfile.txt original as backup,
  3. Use the Cumulus 1 editor (in Edit menu, select Dayfile.txt screen), and let it edit the original file
  4. Delete any days with partial information (e.g. from Cumulus versions that created fewer fields)
  5. Click the Create Missing button
  6. Wait while Cumulus scans all the standard log files, and for each source or derived field in those files, tracks the highest and lowest for each meteorological day that is now missing in dayfile.txt.
    • As an aside, be aware that when the contents of today.ini was used to create a new line in dayfile.txt originally, the high and low that were copied across were derived from all source values that Cumulus obtained from your weather station (depending on your weather station type this might be around every minute; some weather stations provide new readings less frequently, some provide them more frequently).
    • Continuing this aside, the high and low that Cumulus is now deriving are based purely on those values stored in the standard data log, by default that is just every 10 minutes.
  7. The new lines that Cumulus 1 adds to your dayfile.txt are also added to another log file dayfileeditlog.txt, so we can access that to find the new values.
  8. For simplicity, click OK to save the edited dayfile.txt file, and exit the editor.
  9. Rename the amended dayfile.txt as "dayfile(generated).txt".
  10. Create an empty text file in a suitable temporary directory, and name it "dayfile.txt",
  11. Use an external editor to open your empty file.
    • You can use a Comma Separated Value file editor, although you will find this difficult starting with an empty file
    • You can use an editor designed for computer programmers or developers that can edit text file
    • You can use a spreadsheet application, such as "Libre Office Calc" or "Microsoft Excel", providing you obey the editing rules defined later
  12. Now merge in (as read-only text) the previous contents of the original dayfile.txt from the copy taken in step 2
  13. Now merge in (where there are fields missing) just those fields from the respective lines in "dayfile(generated).txt".
  14. Finally, copy your temporary "dayfile.txt" into the "Cumulus\data" folder, so from now on Cumulus accesses the file with the maximum number of fields present.

One note of caution:

  • If a required source or derived value is not present in the standard data log lines, Cumulus will write into related dayfile.txt fields zero for values, and "00:00" for time stamps, because it does not understand the concept of "Null".
  • Cumulus 1 does not have the functionality to retrospectively calculate derived values when it is reading source values in the standard data log line.


Important Rules when editing dayfile.txt

If you are editing dayfile.txt outside the Cumulus 1 or MX software, there is the danger of changing something that prevents Cumulus from understanding the file when it next tries to use it.

General Editing Rules

  1. Take a copy of the file that can be reverted to if there is a subsequent problem, and you have messed up the file that Cumulus (1 or MX) is now trying to use.
  2. Take another copy and use that for your editing, don't edit the actual file being used by the software.
    • This prevents any conflicts between access by the software and access by your script or tool being used to modify the file.
    • It also means that you can go back to the last working copy, you can't upset your "revert" copy.
  3. The file must never be edited with a word processor, as they store many control and identification characters that prevent Cumulus correctly reading the values.
    • It is recommended that you use either a specialised "Comma Separated Value" file editor or a text editor, both of these can be easily used.
      • These tools have the advantage that they can cope with different lines having a different number of fields depending on which version number of Cumulus created each line.
    • You can use a spreadsheet tool, but if you do, there may be a number of settings to change from their defaults to ensure the file remains in a readable format for Cumulus.
      • If you do use a spreadsheet, extra field separators may be added at end of shorter lines as these make all lines end up with same number of fields.
    • Don't remove any figures from fields where figures currently exist, only change one entry into another entry in same format.
  4. Cumulus does not accept the concept of nulls, there is nothing that can be placed as a place-holder when the correct figure is not known, and empty fields are not permitted.
  5. All figures must be within the range of sensible figures for that field (no hour 24 or higher, no signed numbers when accepted values must be positive, don't put in 200 for a relative humidity)
  6. Make sure that any editing does not create any blank lines in the file. Cumulus assumes an empty line means end of processing.
  7. Don't add a header line to the file, Cumulus expects all lines to be data lines.

File specific Editing Rules

Open office (editing cumulus log files).png
  1. The file should be saved without "Byte Order Mark", specialised text editors will include a menu where you select the encoding and can select not to include BOM.
  2. All rows must start with date and include at least 14 further fields in correct sequence.
  3. The (meteorological) date format uses two digits for the year.
    • This is one reason why you need to edit this file using an editor that treats all fields as text (a text editor, a CSV editor, or a spreadsheet program that can be instructed not to recognise special field types).
    • For spreadsheet tools (e.g. Calc in Libre Office, or Microsoft's Excel) avoid using default of recognising formats, ensure that such recognition is turned off (see image), as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to change it to four figure years, and then Cumulus will no longer be able to use the log file.
  4. Remember the month must be the middle figure in the date, USA convention cannot apply within this logfile.
  5. The separator between the three parts of the date should be a '-' hyphen or a '/' slash, it cannot be a space.
    • Whether Cumulus expects a hyphen or a slash is determined by the locale, you must keep to the same locale for the whole file, you cannot change the locale when you do an edit, nor when you update the device running Cumulus.
    • Although, use of comma or point for separating parts of the date is in some locales, and therefore allowed by Cumulus, those locale settings are not recommended as these date separators can cause issues for subsequent edits.
    • If you move your software (any flavour) to a new device, or you change from Cumulus 1 to Cumulus MX (or back), then you must ensure your dates still use the same separator, so all lines are consistent.
  6. Each of the fields from date to the end of the line are separated using the list separator (e.g. a comma or semi-colon) defined for your device. After your editing it must still match what your existing dayfile.txt uses.
    • If you wish to use Excel, or to use "Calc" in 'Apache Open Office', "Libre Office", or similar, you may on opening the file need to pre-select the field separator that is being used now (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" (or whatever similar feature name your tool uses) unselected. Again third party packages processing dayfile.txt will need to recognise your field separator, and some may need to specify it. Don't forget to also select it when you save the edited file (you probably need to select "save as" or the equivalent in your tool to see the option).
  7. Rows can vary in length but only by missing off fields at the end. The minimum number of fields after the date is 14, the maximum varies between different versions.
  8. Each field has a pre-defined format, and the same format must always be used in that field position.
  9. No fields will accept letters.
    • Some fields (e.g. bearings, solar, humidity) are integers (see #List_of_fields_in_the_file) only take integers. Decimals are not allowed in an integer field, so no comma or full-stop can be within these fields.
    • Most value fields are in real number format using your system/locale decimal notation ("x.y" or "x,y"). Trailing zeroes are not required, so you can put an integer in a real number field, you don't have to have a decimal comma or decimal point.
  10. Although only the date and 14 other fields are mandatory, you cannot skip some fields defaulting them to null is not allowed, so you cannot add fields at the end, without adding all earlier fields.
  11. when you do add fields beyond the 14, or however many already exist, be aware that for most derivatives what you add will represent a lowest or highest value and that must be paired with a time-stamp in the next field.
    • Cumulus will only accept highest/lowest figures if each value has any related time-stamp.
  12. Time stamp fields must always be in format HH:mm i.e. 2 digit hour in 24-hour format, followed by a colon, then 2 digit minutes
      • Be aware you will have problems if you, or your editing software, add seconds.
    • If when editing, you don't know what time to quote, the convention is to use a time-stamp of your roll over time i.e. 00:00, 09:00, or 10:00, if you have not looked up the precise time.
    • Except for wind gust (start of line) where an extra field is fitted in, each time field will immediately follow the value field for that parameter.
  13. Shorter lines can have multiple field separators added at end of row added either when editing within Cumulus or when editing using a spreadsheet tool.
    • Nulls (2 field separators without something between them ',,') are thus allowed at end of line, but are not allowed within the part of the line with values and time-stamps.
    • If you are editing out rogue values and if you do not know the value for a particular field within the line, then type in a zero or 9999 for nulls in integer format and an extreme with opposite value (e.g. -999.9 for a signed decimal maximum, and 9999.9 for a decimal minimum) for nulls in decimal format (replace the full stops with your decimal separator).
    • Beware - if you do insert zero or an obviously wrong extreme value, Cumulus will display those in any editing screen where you wish to update the all-time, monthly-all-time, this month, or this year, extremes. This can make editing by picking values in logs harder.
  14. Cumulus itself will use zero for any parameters (e.g. solar) not provided by your station, and will repeat the last valid value if the station fails to send a value it should provide, so if a station fails to send a value for more than a day, dayfile.txt may show the same value as the previous day.
    • Note that Cumulus will stop if your station fails to send what it considers as a vital reading, like pressure or temperature, so the previous point does not apply in all cases.
  15. The row terminator for Windows is CR LF (a combination of Carriage Return represented in many character set encodings by the binary equivalent of decimal 13), and Line Feed represented in many character set encodings by the binary equivalent of decimal 10), ensure any external editor does not change the two character terminator into a single character. Similar rules apply for single character terminators used by other operating systems (Linux uses LF only, Mac uses CR only), don't let any editor you use change a single character terminator into its double character terminator.
    • Problems with terminating characters are intercepted by operating system, before it reaches the software, but may still stop the software understanding the resulting file, so be careful if you edit the file on a different device to that running Cumulus.