Cumulus Wiki

Dayfile.txt: Difference between revisions

Page Discussion

Dayfile.txt (view source)

Revision as of 07:14, 27 September 2022

3,791 bytes added ,  27 September 2022
m
Correcting Typos (mostly missing, but critical, colon characters in links), and mention of some more release 3.20.0 changes
m (→‎Table listing Fields: release 3.20.0)
m (Correcting Typos (mostly missing, but critical, colon characters in links), and mention of some more release 3.20.0 changes)
 
As part of a redevelopment of Wiki, this page has been simplified, by moving some content to new pages. Old links in the support forum, that were to content no longer on this page, will bring you here.
 
'''Content previously on this page has been moved as follows''':
* Explanations about Cumulus terminology can be accessed from [[:Category:Terminology]] page
* Each line in this file represents one day, but the start time need not be same for all fields,
** If you import historic data from before you started using Cumulus into [[Standard log files]],
** then see [[Calculate Missing Values]] page for how to generate new lines in dayfile.txt
* Cumulus MX is more fussy, than Cumulus 1, about various formatttingformatting issues, see below, also see [[Migrating_from_Cumulus_1_to_MX#dayfile.txt]] section for more advice
 
The content that remains on this page is summarised by the table of contents that follows.
* This Wiki page describes one of the [[:Category:Cumulus Files|files]] not included in any release download.
* This daily summary file (Steve Loft shortened that to "day" plus "file") uses a ".txt" extension, and is essentially a plain text file in CSV format.
** CSV stands for "Comma Separated Values", but in the Cumulus context the field separator can be other characters including ";" or "|"
* The file is created in [[Datadata folder|data sub-folder]] of your Cumulus installation when Cumulus needs to store its first line in this file.
* Cumulus MX reads the whole of this file wheninto an internal array (stored in RAM) each time you restart the software,
** So if you move house, it is probably best to archive the old "dayfile.txt", and avoid any discontinuities in any graphs plotted from active file▼
** Please see [[Amending_dayfile#Warning_if_editing_outside_MX|here for implications]]
▲** SoTherefore ifMX youusers who move house,to ita isdifferent probablylocation best toshould archive the old "dayfile.txt", andto avoid any discontinuities in any graphs plotted from active file
 
 
* Thus it is worth backing up this file, to another device than that running Cumulus, on a regular basis, probably a few times a week, to ensure this precious data is not lost if your device running Cumulus has a malfunction (or is damaged), or an electrical blip (or mistake by you) causes corruption to the original file,
 
Cumulus does periodically copy this file within the installation, but the copies are only kept for a limited time, and are on the same physical storage device, and so your main file is corrupted you might not be able to access the Cumulus back up or might not be able to go far back enough forto anfind uncorrupteda file that has not been corrupted:
* The file is backed up when MX is restarted into a date/time stamped sub-folder of the [[Backup folder|'''backup''' folder]]
*The file is also backed up (to a date/time stamped sub-folder within '''daily''' sub-folder of that backup sub-folder) during the end of day process, but (depending on the release you are running) the back-up copy may, or may not, include a line for the previous day (the line that is appended in the same end of day)
 
Retention of these back ups:
* [[File:Badge vMx.png]] MX only keeps the last 9 of the date/time stamped subfolderssub-folders.
* [[File:Badge v1.png]] Legacy Cumulus 1 only keeps up to 8 of the date/time stamped subfolderssub-folders.
 
== Specific issues for MX ==
 
WARNING: It is important to note that some releases of MX are very fussy about consistency in all lines of '''dayfile.txt'''. Earlier releases tended to have better compatibility with the legacy software. Subsequent releases expected every line to be expressed exactly as specified in the locale. MX in its latest release is trying to cope better with inconsistent date separators, and at the time of writing this there was a proposal for forcing this file to use decimal points (full stops not commas) regardless of locale. Therefore, information here might be incorrect for the release that you are using.
 
Date field: Cumulus 1 (C1) did not care what character (or characters) separated the day, month, and year elements of the date field. MX insists that the same character (or characters), as defined in the locale, is used for all lines in file.
 
Real number fields: MX uses the '''locale''' to decide what character (decimal comma or decimal point) separates integer and decimal parts of numbers. Every line of the file must be consistent in use of integer/decimal separator. C1 used the '''Region''' settings in Microsoft's Control Panel to determine how to store (and output) numbers for you, but (except if ''list separation character'' was comma) was fairly tolerant when reading old data in files. At the time of writing this, it was proposed the next major release will force this file to use decimal points (full stops not commas) regardless of locale. This proposal may or may not happen, the point is that information here might be incorrect for the release that you are using.
 
Time-stamp fields: MX insists on HH:mm format being consistently used in every line of file, C1 did not care what (non-space, non-field separator) symbol separated the minutes from the hours.
Number of fields: The number of fields in the file increases in various versions as shown in [[#List_of_fields]].
* C1 will accept (as did Cumulus 2) any line with 15 to 45 fields in it. This is because the earliest version only stored 15 fields, as C1 was developed fields were added (as shown in field list) until there were 45 fields.
* Each release of MX tooexpects a particular number of fields in each line of file; MX has added fields as the software has developed, and it even (as shown in field list) inadvertently added 2 fields later removed. The difference is that MX (for some functionality from release 3.4.5, for all functionality from release 3.9.5) reads the whole file into an internal (RAM held) array with a fixed number of elements, therefore every line ends up with same number of fields when written back into file. If you have some lines in your file that were created by an earlier release of Cumulus, and so have less than whatever is the current number of fields for the release you are using, you can add the missing derived fields, please see [[Calculate Missing Values]] page. Please ensure you use the right version of the "Create Missing" utility mentioned on that page as it also get upgraded when fields are added.
** If you have some lines in your file that were created by an earlier release of Cumulus, and so have less than whatever is the current number of fields for the release you are using, there is a utility to run that will create a new dayfile.txt with all fields populated, please see [[Calculate Missing Values]] page. Please ensure you use the right version of the "Create Missing" utility mentioned on that page as it also get upgraded when fields are added, and it uses various software libraries contained within .dll files specific to a particular CumulusMX.exe release.
 
==How to view or edit this file==
 
ForThe detailed information, pleasehas seebeen moved to [[Amending dayfile|viewing/editing '''dayfile.txt''' page]].▼
An editor has been included within Cumulus:▼
 
* [[File:Badge vMx.png]] From release 3.4.5 (13 Mar 2020) onwards: In [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|the interface]] go to '''Data logs''' menu and select ''Dayfile''▼
▲An editor has been included within recent Cumulus releases of each flavour:
* [[File:Badge v1.png]] From version 1.9.2 (5th October 2011) to final legacy release: On Main Screen from [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|'''Edit''' menu]] select ''Dayfile.txt''▼
▲* [[File:Badge vMx.png]] FromAvailable from release 3.4.5 (13 Mar 2020) onwards: In [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|the interface]] go to '''Data logs''' menu and select ''Dayfile''
▲* [[File:Badge v1.png]] FromAvailable from version 1.9.2 (5th October 2011) to final legacy release: On Main Screen from [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|'''Edit''' menu]] select ''Dayfile.txt''
**'''Note for obsolete version 1.9.0 to 1.9.3:''' There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.
**'''Note for obsolete version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause lines to be stored in incorrect date order (dayfile.txt uses dd.mm.yy, dd/mm/yy or dd-mm-yy, for its date field; and all lines should be in ascending chronological order)
**There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1101. Build 1099 is the standard stable final release of Cumulus 1 for most weather station types, 1100 and 1101 are for specific weather station types.
 
** Please note that the editor in Cumulus 1 allows you to change the date, as well as all the other fields, although the lines must be kept in ascending date order to avoid errors when subsequently reading the file.▼
▲For detailed information, please see [[Amending dayfile|viewing/editing '''dayfile.txt''']].
 
The MX editor does not let you change the date, you can only change other fields or delete an entire line, so you have no control over the order of the lines in the file.
 
 
==Reading the file==
See [[Daily Summary]] page for a full discussion of ways, external to Cumulus, to read this file.
 
Apart from the viewing/editing options just described, Cumulus software reads this daily summary file in variousvaCumulus 1 does not actually number lines, however it does count lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number (numbering from 1) where it first found an error (line numbering uniquely identifies the line, even if a date is duplicated, or a line feed has been deleted so two lines are merged).rious other circumstances depending on the release you are running.
* [[File:Badge v1.png]] The legacy Cumulus has a number of [[Cumulus_Screenshots#View_data|screens for viewing data for various periods]], these use several of the [[Category:Cumulus Files|files]] for their source, including "dayfile.txt". The '''Select a graph''' feature also uses several of the [[Category:Cumulus Files|files]] for their source, including "dayfile.txt". Thus the legacy software only reads the file when the Cumulus user makes a specific request, in normal operation the existing content is ignored, and the end of day action uses a simple "append" instruction.
* [[File:Badge vMx.png]] From release 3.9.2 - b3097 (7 Dec 2020), MX reads the entire '''dayfile.txt''' file, to drive the [[Highcharts_-_Historic|'''historic charts functionality''']]; both in the [[MX Administrative Interface|interface]], and the example included [[New Default Web Site Information|Historic Charts web page]]. ▼
** Consequently, any error in say the date field (or change of separating character) will stop historic charts working.
 
* [[File:Badge v1.png]] The legacy software only reads the file when the Cumulus user makes a specific request, in normal operation the existing content is ignored:
** The legacy Cumulus has a number of [[Cumulus_Screenshots#View_data|screens for viewing data for various periods]], to populate these screens Cumulus has to read all of the [[:Category:Files with Comma Separated Values|Files with Comma Separated Values]] as the derivatives shown generally do not match the extremes tracked in [[:Category:Ini Files|.Ini files]].
** The '''Select a graph''' feature also uses all of the [[:Category:Files with Comma Separated Values|Files with Comma Separated Values]].
** The end of day action uses a simple "append" instruction, so it does not need to read the file
▲* [[File:Badge vMx.png]] FromAs release 3.9.2 - b3097 (7already Dec 2020)mentioned, MX reads the entire '''dayfile.txt''' file when it is first started into an internally held array. That internal copy was introduced in release 3.9.2 - b3097 (7 Dec 2020), it was needed to drive the [[Highcharts_-_Historic|'''historic charts functionality''']]; both in the [[MX Administrative Interface|interface]], and the example included [[New Default Web Site Information|Historic Charts web page]].
** From release 3.20.0 - b3200 (21 Aug 2022) there is a '''This Period Highs and Lows''' web page (accessed from ''Records'' menu) which is similar to the ''screens for viewing data for various periods'' except that the extremes it shows match those tracked in [[:Category:Ini Files|.Ini files]].
** From release 3.20.0 - b3200 (21 Aug 2022) there is an option to '''Reload dayfile''' in a ''Utils'' menu, this updates the internal held array by reading the file again; essential if an edit has been made to the file.
 
==Line numbering and Error messages==
 
** Cumulus 1 does not actually number lines, however it does count lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number (numbering from 1) where it first found an error (this uniquely identifies the line even if a date is duplicated or a line feed has been deleted so two lines are merged).▼
 
Cumulus MX stores the contents of the file in an internal two dimensional array. The outer dimension is identified by numbering (from zero) what were lines in the file. The inner dimension has that line number as element zero, and the fields taken from file are therefore numbered from one.
 
Obviously, line numbering uniquely identifies the line, even if a date is duplicated, or a line feed has been deleted so two lines are merged in the file. This is why you will see line numbers quoted in any error messages related to this file.
 
Some of the editors you may use to view/edit the file may number the lines (in some cases always, in others as an option). If you have seen an error message, then to correct the file you need to use a tool that does number the lines in order to find where the correction is needed. Error messages report just the first error found, so check subsequent lines in case they have same error.
 
= List of fields in dayfile.txt =
 
'''For your installed build please see ''[[dayfileheader.txt]]''. (storedThis withinfile theidentifies folderall thatthe containsfields yourin Cumuluseach executable),line asof thatthe will'''dayfile.txt''' listfile whichin fieldsthe yourrelease Cumulusyou installationare usesrunning.'''
 
That file has changed location from release 3.20.0 - b3200 (21 Aug 2022), it is now in [[data folder]], for Cumulus 1 and all earlier MX releases it was in same directory as the Cumulus executable.
If you have been using Cumulus for a while, you may wonder which of your log file lines might be shorter, so the table below shows fields grouped by the Cumulus version when those fields were added.
 
As explained earlier, the number of fields in a line has (with one exception when fields were added in error) increased as the Cumulus software has been developed. The table below therefore groups the fields under labels that indicate which version/release introduced them.
 
==Field numbering==
 
The table below was first created when the only flavour of Cumulus software available was what we now call the legacy Cumulus 1. Back then the fields '''were''' numbered starting from zero. This reflected the way fields were identified when processed as elements of an single-dimension array variable representing a line.
Please note the list of fields has been rewritten especially for MX. As part of the rewrite, the fields have been renumbered, in some forum posts you might see references to old numbering, in others to new numbering.▼
* The original table below was for Cumulus 1 and then field number '''was''' starting from zero.
▲** Cumulus 1 does not actually number lines, however it does count lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number where it first found an error (this uniquely identifies the line even if a date is duplicated or a line feed has been deleted so two lines are merged).
▲** Please note that the editor in Cumulus 1 allows you to change the date, as well as all the other fields, although the lines must be kept in ascending date order to avoid errors when subsequently reading the file
*The fields are now numbered starting from 1 to fit in with Cumulus MX where when the log file is read, the processing code adds a line number in front of the date field on each line it holds.
** The Cumulus MX user may not be aware of this happening as it is within the internal workings, where data from the file is transferred to an array, or data from the array is written back into the file.
**By using line numbers, MX is able to identify which line has been deleted or edited, MX coding treats the date as a fixed bit of text (MX does not allow you to change a date)
 
The old numbering from zero had two advantages:
# It stressed that the date field was different to the rest, all other fields were either values or time-stamps
# Numbering starting from zero is consistent with standard indexing used for arrays in programming languages (like JavaScript), so the number shown '''was''' the number to quote in any scripts where a Cumulus 1 line was converted to an array, and you needed to address a single field.
 
▲Please note the list of fields has been rewritten especially for MX. As part of the rewrite, the fields have been renumbered, in some forum posts you might see references to old numbering, in others to new numbering. As explained [[#Line numbering and Error messages|above]] the fields are now numbered starting from 1 to fit in with an internal array created when the log file is read, the processing code adds a line number in front of the date field on each array 'line' it holds.
 
== Information shown in the table ==
 
* The date '''mustis supposed to''' be a unique identifier, the same date should not be repeated in another line, however Cumulus 1's editor allows you to change that date field.
** The dates, as mentioned earlier, should be in ascending order.
* The remaining fields were all either numerical values, or a time paired with (except for first wind field) preceding numerical value.▼
** Please see [[amending dayfile]] page for more information on dates
** Cumulus 1 actually enforces this pairing (i.e. it validates that a time is present where it is needed).▼
** Both Cumulus 1 and MX may get confused if you restart the software close to rollover time, and in that confused state they may create a second line with same date as a date already in the file.
*The alphabetic column identifiers used by many spreadsheets are shown▼
** If you regularly restart Cumulus close to rollover time, then a whole sequence of dates can be repeated either as a block or as pairs.
** IMPORTANT: Ensure '''all columns are set to "text" format''', ▼
▲* The remaining fields wereare all either numerical values, or a time paired with (except for first wind field) preceding numerical value.
** Note that you will corrupt this file if you let your spreadsheet recognise content as dates or time, change the number of decimal places.▼
▲** Cumulus 1 actually enforces this pairing (i.e. it validates that a time is present where it is needed). Cumulus MX only validates that each field is of correct type, it does not recognise any relationships between fields.
* The type of field is shown in the table, ▼
▲*The alphabetic column identifiers used by many spreadsheetsspreadsheet editors are shown in the table
** you must not include a sign for an unsigned field, ▼
▲** IMPORTANT: EnsureIf you do use a spreadsheet editor, do ensure '''all columns are set to "text" format''',
** you can not specify a decimal point in an integer field, ▼
▲** Note that you will corrupt this file if you let your spreadsheet recognise content as dates or time, or let it change the number of decimal places.
** all time fields must use 5 character "HH:mm" format
▲* The type of field is shown in the table, :
▲** Note you must not include a sign for an unsigned field,
▲** Note you can not specify a decimal comma/point in an integer field,
** Note all fields of type time must use 5 character "HH:mm" format, although Cumulus 1 will accept other separator characters where ":" shown.
* The field description is shown, together with references to where that terminology is explained
 
|-
|0
|colspanColspan="34" style="background: pink;"|For internal MX purposes, the zero field identifies a field that holds the line number. It is not actually stored as a field in the log file, but precedes any line exchanged via an application programming interface, and therefore is also included in an array representing all the fields in any log file line.
 
If you are processing this log file using a third party (or your own) script, that probably does not place the line number into any array, and your array elements will start at 0 for the field labelled 1 in this table, so putting all field numbers out by 1.
Sfws
5,838

edits

Retrieved from "https://www.cumuluswiki.org/a/Special:MobileDiff/10741"