Difference between revisions of "Dayfile.txt"

From Cumulus Wiki
m (Introduction)
m (Specific issues for MX: Time-stamp issue added)
 
(One intermediate revision by the same user not shown)
Line 23: Line 23:
 
An editor has been included within Cumulus (look up the linked pages for screen shots):
 
An editor has been included within Cumulus (look up the linked pages for screen shots):
 
* [[File:Badge vMx.png]] From release 3.4.5 (13 Mar 2020) onwards:  In [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|the interface]] go to '''Data logs''' menu and select ''Dayfile''
 
* [[File:Badge vMx.png]] From release 3.4.5 (13 Mar 2020) onwards:  In [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|the interface]] go to '''Data logs''' menu and select ''Dayfile''
 +
* Cumulus 2 has no viewing option
 +
* [[File:Badge v1.png]] From version 1.9.2 (5th October 2011) to final legacy release: On Main Screen from [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|'''Edit''' menu]] select ''Dayfile.txt''
 
**'''Note for obsolete version 1.9.0 to 1.9.3:''' There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.
 
**'''Note for obsolete version 1.9.0 to 1.9.3:''' There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.
 
**'''Note for obsolete version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause lines to be stored in incorrect date order (dayfile.txt uses dd.mm.yy, dd/mm/yy  or dd-mm-yy, for its date field; and all lines should be in ascending chronological order)
 
**'''Note for obsolete version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause lines to be stored in incorrect date order (dayfile.txt uses dd.mm.yy, dd/mm/yy  or dd-mm-yy, for its date field; and all lines should be in ascending chronological order)
 
**There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1101. Build 1099 is the standard stable final release of Cumulus 1 for most weather station types, 1100 and 1101 are for specific weather station types.
 
**There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1101. Build 1099 is the standard stable final release of Cumulus 1 for most weather station types, 1100 and 1101 are for specific weather station types.
* Cumulus 2 has no viewing option
 
* [[File:Badge v1.png]] From version 1.9.2 (5th October 2011) to final legacy release: On Main Screen from [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|'''Edit''' menu]] select ''Dayfile.txt''
 
  
 
For detailed information, please see [[Amending dayfile|viewing/editing '''dayfile.txt''']].
 
For detailed information, please see [[Amending dayfile|viewing/editing '''dayfile.txt''']].
Line 50: Line 50:
  
 
Real number fields: MX uses the '''locale''' to decide what character (decimal comma or decimal point) separates integer and decimal parts of numbers. Every line of the file must be consistent in use of integer/decimal separator. C1 used the '''Region''' settings in Microsoft's Control Panel to determine how to store (and output) numbers for you, but (except if ''list separation character'' was comma) was fairly tolerant when reading old data in files.
 
Real number fields: MX uses the '''locale''' to decide what character (decimal comma or decimal point) separates integer and decimal parts of numbers. Every line of the file must be consistent in use of integer/decimal separator. C1 used the '''Region''' settings in Microsoft's Control Panel to determine how to store (and output) numbers for you, but (except if ''list separation character'' was comma) was fairly tolerant when reading old data in files.
 +
 +
Time-stamp fields: MX insists on HH:mm format being consistently used in every line of file, C1 did not care what (non-space, non-field separator) symbol separated the minutes from the hours.
  
 
Number of fields: The number of fields in the file increases in various versions as shown in [[#List_of_fields]].   
 
Number of fields: The number of fields in the file increases in various versions as shown in [[#List_of_fields]].   

Latest revision as of 11:56, 21 August 2021

Cumulus Version MX SpecificCumulus Version 1 SpecificThis Wiki page applies to both Cumulus flavours.

Introduction

This Wiki page describes one of the files not included in any release download. This daily summary file (its formal title is shortened to "day" plus "file") uses a text extension and is created in data sub-folder of your Cumulus installation when Cumulus needs to store its first line in this file.

The support forum may have links to content that is no longer on this Wiki page, so if the link you followed in the forum takes you to the start of this page, rather than to a particular section, the information you seek may now be on another page. this page is full of links that take you to content once on this page. The content that remains on this page is summarised by the table of contents that follows.

About this file

If you import historic data from before you started using Cumulus into Standard log files, then the daily summary for that imported data can be summarised into dayfile.txt line format (and a dayfile.txt file created/updated) using any of the approaches described on the calculate missing values page of this Wiki.

The single file dayfile.txt can contain lines created over a long period of time (speciallog.txt is another log file that contains all dates in a single file, as do all the .ini files).

How Cumulus Creates and Updates this file

Cumulus reads values supplied by your weather station, converts them to the units you prefer, applies any calibration (multiplier and offset) you have set, and then sees if the resulting value implies any extreme records file needs to be updated (these hold derived values like totals, highest, and lowest). Two of those files are today.ini and yesterday.ini and values from those two files are exacted (see linked Wiki pages for how entries in those files map across to the daily summary file) at the end of a meteorological day to calculate the necessary fields for a new line to be added to this file.

As explained on the "meteorological day" page linked above, if your weather station stores weather data, then Cumulus on restarting can (optionally) read the historic data and catch-up by processing those past readings (as per description in start of previous paragraph), including activating any end of day actions (see today.ini), for the period since it was last running.

How to view or edit this file

An editor has been included within Cumulus (look up the linked pages for screen shots):

  • Badge vMx.png From release 3.4.5 (13 Mar 2020) onwards: In the interface go to Data logs menu and select Dayfile
  • Cumulus 2 has no viewing option
  • Badge v1.png From version 1.9.2 (5th October 2011) to final legacy release: On Main Screen from Edit menu select Dayfile.txt
    • Note for obsolete version 1.9.0 to 1.9.3: There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.
    • Note for obsolete version 1.9.3 only: Create missing might in some cases be affected by a bug in 1.9.3 that can cause lines to be stored in incorrect date order (dayfile.txt uses dd.mm.yy, dd/mm/yy or dd-mm-yy, for its date field; and all lines should be in ascending chronological order)
    • There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1101. Build 1099 is the standard stable final release of Cumulus 1 for most weather station types, 1100 and 1101 are for specific weather station types.

For detailed information, please see viewing/editing dayfile.txt.

You might also find it useful to read Correcting Extremes page for information about fixing other files where the highest/lowest/total recorded for day has been corrupted by rogue values.

Reading the file

See Daily Summary page for a full discussion of ways, external to Cumulus, to read this file.

Apart from the viewing/editing options just described, Cumulus software reads this daily summary file in various other circumstances depending on the release you are running.

Badge v1.png The legacy Cumulus has a number of screens for viewing data for various periods, these use several of the for their source, including "dayfile.txt". The Select a graph feature also uses several of the for their source, including "dayfile.txt". Thus the legacy software only reads the file when the Cumulus user makes a specific request, in normal operation the existing content is ignored, and the end of day action uses a simple "append" instruction.

Badge vMx.png From release 3.9.2 - b3097 (7 Dec 2020), MX reads the entire dayfile.txt file, to drive the historic charts functionality; both in the admin interface, and the example included Historic Charts web page. Consequently, any error in say the date field (or change of separating character) will stop historic charts working.

Specific issues for MX

WARNING: It is important to note that MX is very fussy about consistency in all lines of dayfile.txt.

Date field: Cumulus 1 (C1) did not care what character (or characters) separated the day, month, and year elements of the date field. MX insists that the same character (or characters), as defined in the locale, is used for all lines in file.

Real number fields: MX uses the locale to decide what character (decimal comma or decimal point) separates integer and decimal parts of numbers. Every line of the file must be consistent in use of integer/decimal separator. C1 used the Region settings in Microsoft's Control Panel to determine how to store (and output) numbers for you, but (except if list separation character was comma) was fairly tolerant when reading old data in files.

Time-stamp fields: MX insists on HH:mm format being consistently used in every line of file, C1 did not care what (non-space, non-field separator) symbol separated the minutes from the hours.

Number of fields: The number of fields in the file increases in various versions as shown in #List_of_fields.

  • C1 will accept (as did Cumulus 2) any line with 15 to 45 fields in it. This is because the earliest version only stored 15 fields, as C1 was developed fields were added (as shown in field list) until there were 45 fields.
  • MX too has added fields as the software has developed, and it even (as shown in field list) inadvertently added 2 fields later removed. The difference is that MX (for some functionality from release 3.4.5, for all functionality from release 3.9.5) reads the whole file into an array with a fixed number of elements, therefore every line ends up with same number of fields when written back into file. If you have some lines in your file that were created by an earlier release of Cumulus, and so have less than whatever is the current number of fields for the release you are using, you can add the missing derived fields, please see Calculate Missing Values page. Please ensure you use the right version of the "Create Missing" utility mentioned on that page as it also get upgraded when fields are added.

Why this file should be backed up

As explained here Cumulus only logs periodic spot values as it works through processing the values it processes from your weather station. Therefore the Standard log files may not include any maximum or minimum values supplied by your weather station. However, on a day-by-day basis these should be preserved in this daily summary file.

Thus it is worth backing up this file, to another device than that running Cumulus, on a regular basis, probably a few times a week, to ensure this precious data is not lost if your device running Cumulus has a malfunction (or is damaged), or an electrical blip (or mistake by you) causes corruption to the original file,

Cumulus does periodically copy this file within the installation, but the copies are only kept for a limited time, and so your main file is corrupted you might not be able to go far back enough for an uncorrupted file:

  • The file is backed up when MX is restarted into a date/time stamped sub-folder of the backup folder
  • The file is also backed up (to a date/time stamped sub-folder within daily sub-folder of that backup sub-folder) during the end of day process, depending on the release you are running the back up copy may, or may not, include the line that is appended in the same end of day

Retention of these back ups:

  • Badge vMx.png MX only keeps the last 9 of the date/time stamped subfolders.
  • Badge v1.png Legacy Cumulus 1 only keeps up to 8 of the date/time stamped subfolders.


List of fields in dayfile.txt

For your installed build please see dayfileheader.txt (stored within the folder that contains your Cumulus executable), as that will list which fields your Cumulus installation uses.

If you have been using Cumulus for a while, you may wonder which of your log file lines might be shorter, so the table below shows fields grouped by the Cumulus version when those fields were added.


Information shown in the table

  • The fields are now numbered starting from 1 to fit in with Cumulus MX where when the log file is read, the processing code adds a line number in front of the date field on each line it holds.
    • The Cumulus MX user may not be aware of this happening as it is within the internal workings, where data from the file is transferred to an array, or from the array is written back to the file.
    • By using line numbers, MX is able to identify which line has been deleted or edited ignoring the date (although unlike Cumulus 1, MX does not allow you to change a date)
    • Cumulus 1 does not actually number lines, it counts lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number where it first found an error.
  • The original table below was for Cumulus 1 and then field number was starting from zero. So in some forum posts you might see references to old numbering, in others to new numbering. The old numbering from zero had two advantages:
    1. Cumulus 1 stressed that the date field was different to the rest, as it was used as identifier. The date must be a unique identifier, the same date should not be repeated in another line, however Cumulus 1's editor allows you to change that date field.
    2. The remaining fields were all either numerical values, or a time paired with preceding numerical value. Cumulus 1 actually enforces this pairing (i.e. it validates that a time is present where it is needed).
    • Numbering starting from zero is consistent with standard indexing used for arrays in programming languages (like JavaScript), so the number shown was the number to quote in any scripts where a line was converted to an array, and you needed to address a single field.
  • The alphabetic column identifiers used by many spreadsheets are shown, please remember to select options in your spreadsheet to ensure all columns are set to "text" format, you will corrupt this file if you let your spreadsheet recognise content as dates or time, or change the number of decimal places.
  • The type of field is shown in the table, you must not include a sign for an unsigned field, you can not specify a decimal point in an integer field, all time fields must use HH:mm format
  • The field description is shown, together with references to where that terminology is explained

List of Fields

Field number Spreadsheet column Field type Description
0 For internal MX purposes, the zero field identifies a field that holds the line number. It is not actually stored as a field in the log file, but precedes any line exchanged via an application programming interface, and therefore is also included in an array representing all the fields in any log file line.

If you are processing this log file using a third party (or your own) script, that probably does not place the line number into any array, and your array elements will start at 0 for the field labelled 1 in this table, so putting all field numbers out by 1.

Those fields included below have been in dayfile.txt from the start of Cumulus 1 (Version 1.0, the First release on 27th January 2004).
1 A 8 characters Date as 2 figure day [separator] 2 figure month [separator] 2 figure year - the separator is that set in the windows system short date format (see setup)
2 B Unsigned number Highest wind gust speed
3 C unsigned integer Bearing of highest wind gust
4 D 5 characters Time of highest wind gust
5 E signed decimal Minimum temperature
6 F 5 characters Time of minimum temperature
7 G signed decimal Maximum temperature
Consistency Note: In some cases Minimum comes before Maximum, in other cases Maximum is before Mimum
8 H 5 characters Time of maximum temperature
9 I Unsigned number Minimum sea level pressure
10 J 5 characters Time of minimum pressure
11 K Unsigned number Maximum sea level pressure
12 L 5 characters Time of maximum pressure
13 M unsigned number Maximum rainfall rate
14 N 5 characters Time of maximum rainfall rate
15 O unsigned number Total rainfall for the day
Above here represents the minimum length for every line, a count of 15 items
[There is no record of which version added this next field. The Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software (phpBBB3) on 2 Jun 2008, and started afresh without retaining any previous content. Therefore all announcements about the content of each build prior to version 1.7.9 were lost. All that can be deduced is that it was between versions 1.2.5 and 1.5.1 as these do not appear in the release history issued by Steve Loft. The first mention of it in the new forum was not until December 2008, but that was not about when it was released. A web tag for this variable was added in Build 978 of 1.9.1 beta, which was obviously long after it was first calculated.

Because of that, in "DataEditor.cs" (part of the source code that is compiled into CumulusMX.exe) this addition has "Extended for ???" as a comment]

16 P signed decimal Average temperature for the day
(Wind run was added from version 1.8.4)
17 Q unsigned number Daily wind run
(The next pair of entries were added from version 1.8.9 build 907 (June 2010) as part of a total redesign of how dayfile.txt was implemented in Cumulus 1)
18 R unsigned number Highest Average Wind Speed
19 S 5 characters Time of Highest Avg. Wind speed
(The two pairs of humidity entries were added in October 2010, a v 1.9.0 beta, the exact build number is now lost)
20 T unsigned integer Lowest humidity
21 U 5 characters Time of lowest humidity
22 V unsigned integer Highest humidity
23 W 5 characters Time of highest humidity
(The next two entries were added from version 1.9.0)
24 X (not documented) Total evapotranspiration (Only valid for Davis stations, shows zero otherwise)
25 Y unsigned Total hours of sunshine (only valid if sunshine sensor connected)

Important if rollover time is 9 am or 10 am: Most fields in this file are updated taking information from today.ini. For a non-midnight rollover, then the Sunshine hours reported here is from 00:01 on the calendar date corresponding to the date in the first field of this file, to subsequent midnight, and that end time is 9 or 10 hours before when this file is updated. Meanwhile, the sunshine hours count has been reset and so the figure in today.ini is not what is wanted here. For that reason the sunshine hours reported here are taken from yesterday.ini.

(The next 16 entries were added from version 1.9.1 May 2011)
26 Z signed decimal High Heat index (added to Cumulus in 1.7.11 only as spot value, not stored)
27 AA 5 characters Time of high heat index
28 AB Signed decimal High Apparent temperature
29 AC 5 characters Time of high apparent temperature
30 AD signed decimal Low apparent temperature
31 AE 5 characters Time of low apparent temperature
32 AF unsigned number High hourly rain
33 AG 5 characters Time of high hourly rain
34 AH) signed decimal Greatest wind chill (high wind speed, low temperature) (calculated since version 1.8.3 as spot value, not stored)
35 AI 5 characters Time of greatest wind chill
(The next two pairs for dew point were added in version 1.9.2 beta build)
36 AJ signed decimal High dew point
37 AK 5 characters Time of high dew point
38 AL signed decimal Low dew point
39 AM) 5 characters Time of low dew point
(The next three entries were added in version 1.9.2 Build 1004)
40 AN unsigned integer Today's dominant/average wind direction
41 AO unsigned decimal Heating degree days
42 AP unsigned decimal Cooling degree days
The next two pairs were added in version 1.9.3 build 1036 (these only show valid values if appropriate sensors exist)
43 AQ unsigned decimal High solar radiation
44 AR 5 characters Time of high solar radiation
45 AS unsigned decimal High UV Index
46 AT 5 characters Time of high UV Index
The next two pairs were added in version 3.6.0, 2 more derived values and their times
47 AU signed decimal High Feels Like temperature
48 AV 5 characters Time of high feels like temperature
49 AW signed decimal Low Feels Like temperature
50 AX 5 characters Time of low feels like temperature
The next two pairs were added in version 3.6.12
  • Version 3.6.12 (build 3088) was an emergency release to cure serious problems in previous build 3087. It added the following 4 fields (2 values and their times).
    • The 4 extra fields are left empty in this release, although you can add values and time-stamps using the dayfile editor.
  • From version 3.7.0 the first 2 of these 4 fields are populated, and the last 2 are removed, so I have labelled them as error.
51 AY signed decimal High Canadian Humidity Index or Humidex
52 AZ 5 characters Time of high Humidex
Just confirming that the next 2 fields were included by mistake in an emergency release (3.6.12), and are not included in current nor any other version, so have labelled them as error.
53 (error) BA signed decimal Labelled as Low Humidex, but not used, (appear in 3.6.12, but no other version)
54 (error) BB 5 characters Labelled as Time of low Humidex, but not used, (appear in 3.6.12, but no other version)
The next value was added in 3.12.0
53 (new) BA unsigned decimal Cumulative Chill Hours since start of season

Example of the file

An extract of a few lines of a dayfile.txt

01/08/11,19.3,61,10:22,12.5,06:58,23.8,14:49,1014.26,20:46,1018.83,09:28,0.0,00:00,0.0,17.8,21.6,4.6,10:44,36,14:14,86,01:56,3.56,8.9,23.8,14:49,23.1,14:50,12.3,06:59,0.0,00:00,12.5,06:58,11.3,00:16,6.9,14:34,354,2.0,1.5

02/08/11,16.1,20,16:55,14.7,06:45,24.2,13:54,1013.79,19:13,1015.65,11:14,0.0,00:00,0.0,18.9,13.7,8.0,15:55,42,20:42,85,06:50,2.79,4.9,24.2,13:54,24.3,13:55,15.1,06:40,0.0,00:00,14.7,06:45,14.8,11:59,7.0,21:09,57,1.0,1.7

03/08/11,14.5,36,17:23,14.9,05:50,24.6,14:46,1012.70,18:44,1015.99,08:34,0.0,00:00,0.0,19.4,17.2,4.8,16:04,50,14:38,79,07:04,3.05,5.8,24.6,14:46,25.4,14:47,15.0,05:50,0.0,00:00,14.9,05:50,14.2,20:01,8.9,00:16,32,0.8,1.9

04/08/11,17.7,16,15:43,14.1,06:20,25.3,15:06,1013.08,18:42,1015.31,08:28,0.0,00:00,0.0,20.2,19.4,8.1,14:12,52,18:20,92,06:55,3.30,9.1,25.3,15:06,26.8,14:55,14.9,06:20,0.0,00:00,14.1,06:20,15.8,14:55,12.5,06:25,36,1.0,2.9

05/08/11,16.1,32,12:52,14.2,06:12,22.2,14:07,1013.89,00:01,1016.36,09:43,0.0,00:00,0.0,18.6,21.6,5.2,13:00,62,15:57,87,06:11,3.30,8.4,22.2,14:07,23.5,14:10,14.8,07:19,0.0,00:00,14.2,06:12,15.4,10:33,12.0,06:03,34,0.9,1.3

06/08/11,16.1,309,11:15,14.3,05:29,22.4,17:12,1014.46,20:02,1016.97,10:38,0.0,00:00,0.0,18.4,19.2,5.5,16:21,55,13:33,92,05:20,2.79,7.9,22.4,17:12,23.3,18:17,15.1,06:09,0.0,00:00,14.3,05:29,14.2,18:12,10.9,10:38,32,1.1,1.3

07/08/11,17.7,342,13:24,12.9,05:47,24.1,14:53,1013.92,19:49,1016.43,09:36,0.0,00:00,0.0,18.4,19.1,6.3,14:06,48,12:45,89,05:36,3.30,9.0,24.1,14:53,24.6,15:48,13.3,05:47,0.0,00:00,12.9,05:47,14.6,15:52,10.7,11:33,11,1.6,1.7