5,838
edits
Amending dayfile: Difference between revisions
m
Spelling corrections
m (Spelling corrections) |
|||
|
Cumulus uses a daily summary log file, the fields in that file are listed at [[Dayfile.txt#List_of_Fields]]. The information about amending the file that was originally on the same page has been moved to this page. {{Version badge 1}}When the text was first created (on the other page) it was for the (legacy) Cumulus 1 software. {{Template:Version badge Mx}}As MX was developed, the text here has been amended to keep up, it currently applies up to release 3.12.0.
▲{{Version badge 1}}{{Template:Version badge Mx}}[[Category:Cumulus Files]][[Category:MX txt Files]]
=How Cumulus uses the daily summary log=
For MX release 3.9.2 - build 3097 onwards, when the software first starts, the whole of '''dayfile.txt''' is read, the contents are used to drive the [[Highcharts_-_Historic]] functionality.
The file is also read, if you are using the editor provided in either Cumulus flavour.
==Writing a new line to the daily summary log file==
As discussed in [[Correcting_Extremes]], it is possible for rogue values to be read from a weather station, and propagate into various log files. An error in many of those files, corrupts a particular extreme record (or more than one), but generally does not stop Cumulus working.
Since release 3.9.2 (build 3097) MX now reads the entire daily summary log when you start it up
== Thoughts required ==
To correct '''dayfile.txt''', you might think the best approach is to look in your [[standard log files|log file]] covering the relevant date, or to use "Create Missing" (which works differently for the legacy software and MX as explained later). Often that is not a good idea, your standard log file might be corrupted as well, and since these log files only record spot values, they miss any extremes occurring between the log entries. Equally, to stop Cumulus, and make it rewind, may worsen the problem, because you throw away the good data you have on other derivatives just to try to resolve one rogue value. If you discover the corruption within a few days of it happening, you can make use of an earlier [[Backup folder|back-up]] as explained later.▼
▲To correct '''dayfile.txt''', you might think the best approach is to look in your [[standard log files|log file]] covering the relevant date, or to use "Create Missing" (which works differently for the legacy software and MX as explained later). Often that is not a good idea, your standard log file might be corrupted as well, and since these log files only record spot values, they miss any extremes occurring between the log entries.
Equally, to stop Cumulus, and make it rewind, may worsen the problem, because you throw away the good data you have on other derivatives just to try to resolve one rogue value.
If you discover the corruption within a few days of it happening, you can make use of an earlier [[Backup folder|back-up]] as explained later.
| Moving from one device to another (and not ensuring same locale on both devices)
| Correct "locale" and/or use an external editor that offers "Replace all" (see [[#Validation by in-built editors]])
| Use '''Control Panel''' to correct region (that defines decimal symbol).
|-
| Inconsistencies in list separator (what comes between fields)
| Moving from one device to another (and not ensuring same locale on both devices)
| Correct "locale" and/or use an external editor that offers "Replace all"
| Use '''Control Panel''' to correct region (that defines list separator). See [[#Using the Cumulus 1 editing feature]] to check file is now consistent.
|-
| Duplication of dates between lines (either consecutive lines, or non-adjacient lines)
# Restarting Cumulus after a crash, either manually using "Rewind" approach or Cumulus is confused by a corrupted file
| Manual editing outside MX in a text editor, if particular dates appear twice, see [[#Dates restart/repeat]]
|
|-
| Some lines with fewer fields than others
| As explained at [[Dayfile.txt#List_of_Fields]], as Cumulus has developed, more fields have been added
| Use [[Calculate_Missing_Values#CreateMissing.exe|separate CreateMissing.exe utility]]
| Use workaround described at [[#Legacy Workaround]]
|-
|}
== Editors built into Cumulus ==
==Missing or Corrupted past ''dayfile.txt'' lines in any Cumulus software==▼
If you
* MX: Use [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|Edit menu in MX's admin interface]]
* C1: Use [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|
===Validation by in-built editors===▼
Both the legacy editor and the MX editor will ensure that the correct number of fields is stored (as defined at the release where you do editing) (one common error in an external editor is to accidentally add/delete a field).▼
The legacy editor will allow you to edit the date field, the MX editor cannot change the date field. The MX editor, reads the file into an array, it uses the array index for all actions on a particular line, and it then writes the array back to the file when you finish editing.
The legacy editor
Unfortunately, '''the editor provided with MX does not validate any fields'''. In MX, the editor will save an edited line, even if there are errors in individual fields:
* you can put inappropriate content in a particular field (integer, real number, time-stamp)
* you can use the wrong separator in fields you do edit (i.e. between hour and minute for time-stamps, or between integer and decimal parts in any real number)
▲==Missing or Corrupted past ''dayfile.txt'' lines in any Cumulus software==
If you have one, or more, dates missing in your dayfile.txt file, then the
* '''Has the line been deleted by accident?'''
* ''Is the line missing because it was never saved into the file?''
* If a line for a particular date was present before, but is now corrupted or missing:
*# See if you have a back-up of '''dayfile.txt''' with the line present, and correct
*#* If
*#* If
*# If you have a suitable backup available, '''take a copy of that back-up file'''
*# Append onto the copy of the backup, '''any dates after when that copy ends''', taking the extra lines from the current dayfile.txt
*# Rename the current dayfile.txt to say dayfile.old
*# Rename the copy you have edited to dayfile.txt and place into '''[[Data folder|data]]''' sub-folder
*# Cumulus will now use the file with all days correct
* If Cumulus never saved the line in the file in the first place
*# The missing line will not be in any back-up
*# If it is the last line on the file that is missing (i.e. last rollover failed), take a copy of the whole [[data folder]], and keep that copy in a safe place
*# Now look in [[backup folder]], and open the '''daily''' sub-folder
*# If there is a subfolder within "daily" that was successfully created during the rollover that failed, rewind Cumulus by overwriting the contents of '''data'' folder with files from the backup in '''daily''' sub-folder. Restart Cumulus, and let it create a new dayfile.txt line. Stop Cumulus again, restore the original files (except "dayfile.txt) from the copy you put in safe place
*# If the rollover failure meant a backup was not created in the "daily" sub-folder, this will usually be the case, you need to follow instructions for the MX, or legacy, Create Missing
''One lesson here, is to try to remember (once a week), to check your dayfile.txt log file is okay, because Cumulus retains back-ups for only the last 7 days''
'''Another lesson here, is to periodically take
==Dates restart/repeat==
If you have a Cumulus crash (either because the connection between the weather station and Cumulus fails; or because there is a
If Cumulus was working correctly before the problem, then the lines stored before the problem should be okay, just delete the lines that repeat earlier dates, so the file ends up being date ascending order with no
If there are two lines with the date when the problem
If the dates restart in your daily summary log file, because you manually stopped Cumulus close to a rollover time, or you corrupted a file perhaps by regressing to an older release and back; then it is likely you will need to merge the two lines with same date, deciding for each value field which is more likely to be right, and matching it with correct time-stamp. A rainfall (or wind run) total might require summing totals in the two individual lines, or discarding a rouge value and accepting the other, your judgement!
▲==Validation by in-built editors==
==Correcting date separator errors==▼
▲Both the legacy editor and the MX editor will ensure that the correct number of fields is stored (as defined at the release where you do editing) (one common error in an external editor is to accidentally add/delete a field).
Cumulus uses a format of "day of month in 2 digits", separator, "month number as 2 digits", separator, "last 2 digits of year" in the first field of every line stored in '''dayfile.txt'''. It might be expected that "separator" is consistent throughout the file, because the device/locale when the line is created determines the separator. The problem comes when a user moves their Cumulus installation to a new device, and forgets to ensure that new device uses same (Microsoft Windows PC) region, or (Unix, Linux, and other operating systems) locale, settings.
▲The legacy editor does also validate individual fields for appropriate content (integer, real number, time-stamp).
Historic notes:
*For builds 3000 to build 3049, MX used fixed offsets to find day, month, year in date field. This meant MX could process any file that the legacy software had accepted. The issue was this did not work for locales that used two characters for separator, and with MX now working with far more locales, this proved to be a serious problem.
*From Build 3050 MX uses the separators, defined by the locale, to split the values (so allowing for multi-character separators). The same character(s) had to be used in every line of the file, or this gave a new problem.
Some [[Daily Summary|third party routines]] for reading dayfile.txt take a different approach, they check the first field for the first character that is not a "0 to 9" digit, take that as start of separator, then look for the next "0 to 9" digit, that belongs to next part of the date, so the separator ends at previous character. Other third party routines, ask user what they use as date separator.
▲==Correcting date separator errors==
=== Date Separator in MX===
The legacy software accepts any character (except space) as the separator between the day of month, the month, and the year, elements of the date. Therefore Cumulus 1 does not care if that separator is sometimes "/", sometimes "-", and/or sometimes ".".
▲Historic note: In Cumulus 1 builds, any symbol (including ;, :, /, &, -, and .) character (excluding what is defined as list separator) within the first field of each line was treated as part of separator characters when parsing for date. For builds 3000 to build 3049, MX used fixed offsets to find day, month, year. From Build 3050 MX uses the separators defined by the OS to split the values (so allowing for multi-character separators), but it does mean it will be sensitive to the separator value changing.
= Importing data not recorded by Cumulus =
You might have been using your weather station with some other weather software before you installed Cumulus. If you can get weather data in the format of daily summaries (and the rollover times match), you can import that data into the Cumulus '''dayfile.txt''' file using a script or spreadsheet package. All you have to ensure is that you can arrange the output to be in lines with fields in sequence shown in [[Dayfile.txt#List_of_Fields]]. There is more guidance later on this page about the rules you must obey for this file.
If you have imported the data from the other weather software into the [[Standard_log_files]] format, then
* a separate utility "CreateMissing.exe" can create the new rows, for MX users, for those days previously missing in dayfile.txt, as explained below.
* in the Cumulus 1 editor, ''Create missing'' can insert the new rows, for those days previously missing, in dayfile.txt, as explained below.
= Using CumulusMX.exe editing functionality =
Pick '''Edit''', click that, and an editing dialog pops up (MX uses '''altEditor ''' software for this). The pop up window does not let you change the line number nor the date, but all other fields show their current contents and you can overtype as necessary. Scroll down to see 2 buttons (how they are labelled depends on which version you are using), the left hand button ignores any edits you have made (it is labelled 'Close' or "Cancel" and simply does same effect as clicking the "x" in the top right corner), it prevents the api sending any replace message back to the MX engine. The right hand button saves your changes (even if it is labelled 'Edit' rather than "Save" in the version you are using) by using the api to send the replacement array back to the MX engine where it will replace the relevant line number before writing back to the log file.
There is no validation in the MX editor that was set up relatively quickly in version 3.4.5 as the first of 3 log file editors to plug a gap in MX functionality in earlier versions, so you must manually ensure you enter correct data:
* some fields can only accept integers, other expect decimals,
* and some fields can accept negatives, others don't accept signed numbers
* some fields have a minimum and/or maximum acceptable value
* time-stamps must use a colon between the 24 hour time sections for hours and minutes
As all lines are passed back via an application programme interface to the MX engine, there is no validation there either, the new line replaces the old one when the whole file is recreated. It is likely that the next time MX attempts to read the dayfile.txt it will find any error. There was a third-party proposal to add simple validation, as listed above into a replacement editor,
Pick '''Delete''', click that, and a simple dialog pops up (MX uses '''altEditor ''' software for this) showing all the fields in the selected line and asking you to confirm that you want to delete it. Again, the labelling on the buttons varies depending on which version you are running, one confirms the deletion (which sends the array back to the MX engine with an instruction that line number is to be deleted. Despite the MX engine getting a copy of the fields that are to be deleted, it
==Cautions if using an obsolete MX release==
* '''Cumulus MX beta version 3.0.0 (checked at build 3043) does not provide an editor'''
| |||