Dayfile.txt
This article is about the Daily Summary logging file
Introduction
Cumulus maintains a daily log file that holds the highs and lows of each day, as well as a few other nuggets of information. In all flavours of Cumulus, this file is only updated (with exclusive lock) during the end of meteorological day process. In that process it is also read if the generation of NOAA reports has been requested.
In Cumulus 1 only, the figures contained in the file are used for the 'This period' display accessed from the View menu and to build any graphs based on daily values.
The format of this file is the same for both Cumulus 1 and Cumulus MX, although the number of fields in the file increases in various versions as shown at the end of this article. The file can be ported between flavours only if both are run with exactly the same locale settings, as using a different locale may change the field separator or the symbol used for decimal points.
Changes in different releases of Cumulus
Be aware that this article was written for Cumulus 1, amended to also cover MX 3.0.0. Whilst the list of fields has been kept updated for more recent MX versions, other parts of this article may not be applicable to latest MX version, check for edits by developer.
Note for obsolete version 1.9.0 to 1.9.3: There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.
Note for obsolete version 1.9.3 only: Create missing might in some cases be affected by a bug in 1.9.3 that can cause incorrect date order for records (dayfile.txt uses dd/mm/yy or dd-mm-yy and all records should be in ascending chronological order)
There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1100. Build 1099 is the standard final release of Cumulus 1 as this section was updated.
Cumulus MX v3.0.0 (checked at build 3043) does not provide an editor
Cumulus MX Version 3.4.5 - Build 3069 onwards provides an editor
When Cumulus is left running
- Cumulus is frequently reading observations from your weather station, but these don't affect the daily log "dayfile.txt" as it is only updated once a day.
- There are no updates to dayfile.txt at any other times, but (for Cumulus 1 only) the contents of the file are read and processed for many of the display and edit menu options that can be selected from the main Cumulus 1 screen.
- Cumulus tracks the highs and lows in weather observations by comparing read values against those it has stored in Today.ini, updating that file as required.
- What is stored in today.ini is processed into what get stored as next line in dayfile.txt as described in next section.
*(It also updates Alltime.ini, Monthlyalltime.ini, Year.ini, and Month.ini when appropriate.
- (Periodically, the weather readings and derived values are stored in Standard log files which are set up for each month).
- Cumulus will not mind you accessing the daily log outside its software, except when it needs write access for processing end of day.
- If you do need to correct some rogue data in the log file, first take a copy and work on that copy, because any edits you do could muck up the specific format that Cumulus 1 or MX needs, there is a section on dealing with rogue data below. Only when you are absolutely sure that your edited copy meets all the constraints listed later, should you replace the original.
When Cumulus processes the end of the (meteorological) day
- A new row is appended to dayfile.txt, the values are prepared from reading "today.ini" file, not all values available in "today.ini" are stored in dayfile.txt.
- Some of this information is also stored in yesterday.ini.
- Back ups of both today.ini, and dayfile.txt, log files in their state after the end of day update are copied to the 'cumulus\backup\daily' folder, a maximum of only 9 daily sub-folders are retained.
Optional
- Some people require a copy of the local file to use on their web server. Consequently, after it has been updated they file transfer it to (or if their web server is local, copy it to) their web server. One way of doing this is described here.
- Some people take a copy of the local file, and use it locally for other purposes. See How you can use the daily log section and also the [[[[Category::User Contributions|Cumulusutils]]]] link.
- For some people it is easier to follow an option of converting the file into a database table, and that table having a new row added each time the file gains a new line as described below.
Populating a database table
- The article here describes a method that can be used with Cumulus 1 to mimic the contents of dayfile.txt in a database table. However, be aware that the later versions of that script have bee edited for MX, so you will need to use an older version of the script that fits the version of Cumulus 1 you are using.
UPDATE BELOW
- There is a section at Cumulus MX to explain how MX includes the ability to generate SQL for creating the database table, for updating it with past data, and to add a new row at the end of the day for the standard database table version of this daily summary log.
Using that table
In both cases, your web site can use that database table avoiding any clash of timing with the Cumulus 1 or MX use of the daily summary log.
For examples of some of the third party tools (Cumulus) using the database daily summary table see here. Of course there are also a lot of tools written to use a copy of the dayfile.txt log file, and some of these could be adapted to use the database table instead, if you are a programmer.
In my case I also store the equivalent of what appears on my version of "thismonth.htm" each month in another database table, i.e. I have one database table column for each of the weather derivatives I show on my web page that show this month's values; it is many more derivatives than are shown on the standard web page, but some are initially hidden. Consequentially, when my daily update script detects from the date that it is processing the last day of a month, it then starts another script that reads all the rows in the daily summary table for that month, and stores the highest/lowest/total (as relevant) in my monthly_summary table (nothing to do with the "monthly" table that MX can generate from its standard log file). This monthly summary table allows me to have web pages that compare consecutive months or compare months between years. Just another example of how much you can get from just one log file!
Alternative schemas
Of course you do not need to exactly mimic the log file with the schema in your database table, your weather station may not produce solar values so those fields in dayfile.txt need not be columns in your database table, or you may wish to add other values from external sensors or other log files.
With Cumulus 1, you would need to be a programmer and write your own script to update the database table with your own schema. You might use the importCumulusFile article to start you off.
You might also, as I did, want your script to validate what it reads from the daily summary log to ensure only valid numbers and times are stored in your database table, while any invalid inputs are stored as nulls by your script. In my own case, my daily summary table has no solar columns but it does have several additional columns (including the daily increment of chill hours, the cumulative chill hours, the contents of the Weather Diary, the time of the last rain tip, wind bearings as compass characters (e.g. NNW) as well as numerical bearings). When I used Cumulus 1 I wrote a PHP script to find all these additional values, for example it reads the today.ini and month.ini log files as stored in the end of day backup (not the ones being updated for new day in data folder), and the weather diary in log.xml in data folder.
MX allows you to specify a different schema in the SQL it generates, but it does not offer that validation feature I just mentioned. I continue to use my Cumulus 1 script (with some changes as for example the weather diary works differently, I am querying SQLite from \CumulusMX\data\diary.db) now I use MX. In the MX standard functionality, you are limited to using web tags for your inputs, and some of those are affected by the end of day process. I tried various content in the custom EOD query, but it did not give me what I need for the scripts that produce my web pages.
MX automatically stores all end of month figures as log files, a feature that Cumulus 1 and 2 lacked, but as yet it does not actually use this extra data, and provides no simple facility to put what is in these files into database tables. There is no end-of-month selection for updates in MX, so you can't easily get as much from dayfile.txt as I do.
When Cumulus is restarted after a break in running
- It reads the daily log and uses the rainfall totals for each day stored in the daily summary log to calculate the rainfall for this month, and this year/season (see this Cumulus 1 FAQ)
- Thus you must not have another process attempting access to the daily log when Cumulus is re-starting.
- For Cumulus 1, back ups of 8 selected log files including dayfile.txt that are copied to start-up folders in the 'cumulus\backup' folder, the last 8 start-up folders only are retained.
- For Cumulus MX, there are backups of 10 files, the extra ones are the weather diary and Cumulus.ini, that are copied to start-up folders in \CumulusMX\backup\, again there are only 9 kept, unless you back these up somewhere else.
How you can use the daily summary log file
- If you want to run scripts that use the daily summary log file, it is best if you take a copy first, you can ask Cumulus 1 to take a copy after each update by using the Daily box in the bottom left of the Sites/Options frame within the Internet options screen from the Configuration menu; that will safely take a copy of 'dayfile.txt' after it is updated. This has the advantage it happens even if Cumulus has been stopped and restarted and rollover is happening during catch-up and so not at usual rollover time according to the computer clock. See Cumulus 1 Help for information on using this feature, I add a redirection ">daily_batch.log" in the parameter box alongside so that any output from running the command file I specify in the main box is sent to a log file overwritten in each run; this enables me to see the reason for any failure.
- Cumulus MX has option to list files to be transferred once a day as part of rollover, so you can use that to generate your extra copy. This has the advantage it happens even if Cumulus has been stopped and restarted and rollover is happening during catch-up.
- A third party tool "Cumulus Toolbox" can also be used to copy/transfer files at a particular time. Note this cannot tell whether Cumulus has done its rollover at the normal time, or during catch-up.
- There are other ways to specify that when a file changes it is copied somewhere.
- The system routines that Cumulus uses to access dayfile.txt require exclusive use of that file, so if you have any other process trying to access that file when Cumulus restarts, when Cumulus processes end of the (meteorological) day, or when a relevant option is selected from View or Edit menus, either your external process or the Cumulus process may fail.
- Normally if you use any third-party packages like for example "Cumulusutils", the separator used in first line is assumed to be true for all lines.
- Some third-party tools have to be told what separator you use for dates, before they can read your dayfile.txt.
Cumulus 2
Does not provide any viewing functionality.
Cumulus MX
There is a dayfile editor within the admin interface to edit this log file.
Only from MX version 3.6.0 has this been able to read the log file if it has some lines that were created using Cumulus 1 (with less than 46 fields). In the same version of MX, the number of fields in this log file was increased by 4 when Feels Like temperature was added. From that version all lines viewed in this editor will have 50 fields. The content of any field that was not in the line when it was created will be an empty string as far as this editor is concerned and any line edited and saved, therefore gains all these empty fields and will be stored as 50 fields until version 3.6.10.
From Emergency Version 3.6.12 (formally released in 3.7.0), all lines have 54 fields. The extras are Canadian Humidity Index (Humidex).
For Cumulus MX, when you select a line, both Edit and Delete buttons are enabled. There is no way of inserting new lines into dayfile.txt from within MX, nor of changing the dates in the file.
Editing inside MX
Pick Edit, click that and an editing dialog pops up, that does not let you change the line number nor the date, but all other fields show their current contents and you can overtype as necessary. Scroll down to see 2 buttons (how they are labelled depends on which version you are using), the left button ignores any edits you have made (it is labelled 'Close' or "Cancel" and simply does same effect as clicking the "x" in the top right corner), and the right hand button saves your changes (even if it is labelled 'Edit' rather than "Save" in the version you are using).
Using the daily summary log on your web-site
If you upload the log file to your web site then (with the help of JavaScript) you can read the log file to obtain information to show on a web page. You could have a web page that shows a today.htm like table for the last 7 days by combining reading Cumulus web tags with reading from the log file.
Search the Cumulus support forum to see (for example) how others extract information from dayfile.txt to display on their web page a set of fields similar to those shown for 'Yesterday.htm' web page for other dates in the past, such as one year ago.
If you use a script to read what is in the daily summary log file into a database table, or use the functionality in Cumulus MX to upload automatically to a database table, then see Daily_Summary article for information about ALL of the ways to show values from this database table.
Viewing summary figures for a month or period
These notes apply to Cumulus 1 only
To view a summary of dayfile.txt for a month, calendar year or selected period, use This month (choose any month, default is month from your computer system date), This year (choose any year, default is year from your computer system date), or This period (choose any start and end dates, default is yesterday calculated from your computer system date), within the View menu.
- Remember the daily summary log has its records based on rollover to rollover days.
- In all cases they exclude the today details that are not stored on dayfile.txt until the end of day rollover.
- If you use 9am or 10am rollover, and choose View This period between midnight and your 9am/10am rollover any day your latest meteorological day is the yesterday in terms of your computer system date that 'This period' tries to display as its default day, and the display will initially appear blank.
- If you use 9am or 10am rollover, and choose View This month before your 9am/10am rollover on the first day of a new calendar month your latest meteorological month is different to your computer system month 'This month' tries to display as its default month, and the display will initially appear blank.
Most of the displayed results are for observations in the daily summary log, but a few parameters are not in that log and are derived from the monthly logs (e.g. average wind speed) or the weather diary (e.g. count of days with snow lying).
- On the screen displayed after selecting This month, you can change the month and year required using the options at bottom left, click Update Display and the revised summary will be calculated.
- On the screen displayed after selecting This year, you can change the year required using the options at bottom left, click Update Display and the revised summary will be calculated.
- On the screen displayed after selecting This period, you can change the start date and end date then click Update Display to get the equivalent calculations displayed for part of a month or any other period.
Note differences between observation reports on View screens and those available as web tags.
- Date and time stamps:
- The day number shown on screen is the meteorological day (changing at rollover and that may be at midnight or 9am/10am) as that date appears in dayfile.txt;
- A time-stamp (with time and date) given in a web tag quotes a calendar date (always changing at midnight).
- Reported statistics example:
- The screen shows total number of dry or wet days in the month;
- The web tags report longest dry or wet period in the month.
List of fields in dayfile.txt
Variation by Cumulus version
The dayfile.txt has grown as Cumulus's functionality has been extended, the table below shows fields grouped by the Cumulus version when those fields were added.
If you have been using Cumulus for a while, then some lines will be shorter than others, the number of fields per line growing when a new release adds new fields, Cumulus does not in normal operation modify earlier lines, but both Cumulus 1 and MX provide editors where it is possible to modify any line.
There is information earlier in this article about how you might be able to recalculate values to put in for fields that did not exist when any particular line was created.
For your installed build please see dayfileheader.txt (stored within the folder that contains your Cumulus executable), as that will list which fields are available for you.
Information shown in the table
- The fields are now numbered starting from 1 to fit in with Cumulus MX where when the log file is read, adds a line number in front of the date field.
- The Cumulus MX user may not be aware of this happening as it is within the internal workings.
- The original table below was for Cumulus 1 and then field number was starting from zero. So in some forum posts you might see references to old numbering, in others to new numbering. The old numbering from zero had two advantages:
- It stressed that the date field was different to the rest. The date must be a unique identifier, the same date should not be repeated in another line.
- The remaining fields were all either numerical values or a time paired with preceding numerical value.
- Numbering starting from zero is consistent with standard indexing used for arrays in programming languages (like JavaScript), so the number shown was the number to quote in any scripts where a line was converted to an array, and you needed to address a single field.
- The alphabetic column identifiers used by many spreadsheets are shown, please see warnings about using spreadsheets for editing earlier on this page
- The type of field is shown, you must not put a sign for an unsigned field, you can not specify a decimal point in an integer field, all time fields must use HH:mm format
- The field description is shown, together with references to where that terminology is explained
List of Fields
| Field number | Spreadsheet column | Field type | Description |
|---|---|---|---|
| 0 | For internal MX purposes, the zero field identifies a field that holds the line number. It is not actually stored as a field in the log file, but precedes any line exchanged via an application programming interface, and therefore is also included in an array representing all the fields in any log file line.
If you are processing this log file using a third party (or your own) script, that probably does not place the line number into any array, and your array elements will start at 0 for the field labelled 1 in this table, so putting all field numbers out by 1. | ||
| Those fields included below have been in dayfile.txt from the start of Cumulus 1 (Version 1.0, the First release on 27th January 2004). | |||
| 1 | A | 8 characters | Date as 2 figure day [separator] 2 figure month [separator] 2 figure year - the separator is that set in the windows system short date format (see setup) |
| 2 | B | Unsigned number | Highest wind gust speed |
| 3 | C | unsigned integer | Bearing of highest wind gust |
| 4 | D | 5 characters | Time of highest wind gust |
| 5 | E | signed decimal | Minimum temperature |
| 6 | F | 5 characters | Time of minimum temperature |
| 7 | G | signed decimal | Maximum temperature |
| 8 | H | 5 characters | Time of maximum temperature |
| 9 | I | Unsigned number | Minimum sea level pressure |
| 10 | J | 5 characters | Time of minimum pressure |
| 11 | K | Unsigned number | Maximum sea level pressure |
| 12 | L | 5 characters | Time of maximum pressure |
| 13 | M | unsigned number | Maximum rainfall rate |
| 14 | N | 5 characters | Time of maximum rainfall rate |
| 15 | O | unsigned number | Total rainfall for the day |
| Above here represents the minimum length for every line, a count of 15 items | |||
| [There is no record of which version added this next field. The Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software (phpBBB3) on 2 Jun 2008, and started afresh without retaining any previous content. Therefore all announcements about the content of each build prior to version 1.7.9 were lost. All that can be deduced is that it was between versions 1.2.5 and 1.5.1 as these do not appear in the release history issued by Steve Loft. The first mention of it in the new forum was not until December 2008, but that was not about when it was released. A web tag for this variable was added in Build 978 of 1.9.1 beta, which was obviously long after it was first calculated.
Because of that, in "DataEditor.cs" (part of the source code that is compiled into CumulusMX.exe) this addition has "Extended for ???" as a comment] | |||
| 16 | P | signed decimal | Average temperature for the day |
| (Wind run was added from version 1.8.4) | |||
| 17 | Q | unsigned number | Daily wind run |
| (The next pair of entries were added from version 1.8.9 build 907 (June 2010) as part of a total redesign of how dayfile.txt was implemented in Cumulus 1) | |||
| 18 | R | unsigned number | Highest Average Wind Speed |
| 19 | S | 5 characters | Time of Highest Avg. Wind speed |
| (The two pairs of humidity entries were added in October 2010, a v 1.9.0 beta, the exact build number is now lost) | |||
| 20 | T | unsigned integer | Lowest humidity |
| 21 | U | 5 characters | Time of lowest humidity |
| 22 | V | unsigned integer | Highest humidity |
| 23 | W | 5 characters | Time of highest humidity |
| (The next two entries were added from version 1.9.0) | |||
| 24 | X | (not documented) | Total evapotranspiration (Only valid for Davis stations, shows zero otherwise) |
| 25 | Y | unsigned | Total hours of sunshine (only valid if sunshine sensor connected) |
| (The next 16 entries were added from version 1.9.1 May 2011) | |||
| 26 | Z | signed decimal | High Heat index (added to Cumulus in 1.7.11 only as spot value, not stored) |
| 27 | AA | 5 characters | Time of high heat index |
| 28 | AB | Signed decimal | High Apparent temperature |
| 29 | AC | 5 characters | Time of high apparent temperature |
| 30 | AD | signed decimal | Low apparent temperature |
| 31 | AE | 5 characters | Time of low apparent temperature |
| 32 | AF | unsigned number | High hourly rain |
| 33 | AG | 5 characters | Time of high hourly rain |
| 34 | AH) | signed decimal | Greatest wind chill (high wind speed, low temperature) (calculated since version 1.8.3 as spot value, not stored) |
| 35 | AI | 5 characters | Time of greatest wind chill |
| (The next two pairs for dew point were added in version 1.9.2 beta build) | |||
| 36 | AJ | signed decimal | High dew point |
| 37 | AK | 5 characters | Time of high dew point |
| 38 | AL | signed decimal | Low dew point |
| 39 | AM) | 5 characters | Time of low dew point |
| (The next three entries were added in version 1.9.2 Build 1004) | |||
| 40 | AN | unsigned integer | Today's dominant/average wind direction |
| 41 | AO | unsigned decimal | Heating degree days |
| 42 | AP | unsigned decimal | Cooling degree days |
| The next two pairs were added in version 1.9.3 build 1036 (these only show valid values if appropriate sensors exist) | |||
| 43 | AQ | unsigned decimal | High solar radiation |
| 44 | AR | 5 characters | Time of high solar radiation |
| 45 | AS | unsigned decimal | High UV Index |
| 46 | AT | 5 characters | Time of high UV Index |
| The next two pairs were added in version 3.6.0, 2 more derived values and their times | |||
| 47 | AU | signed decimal | High Feels Like temperature |
| 48 | AV | 5 characters | Time of high feels like temperature |
| 49 | AW | signed decimal | Low Feels Like temperature |
| 50 | AX | 5 characters | Time of low feels like temperature |
The next two pairs were added in version 3.6.12
| |||
| 51 | AY | signed decimal | High Canadian Humidity Index or Humidex |
| 52 | AZ | 5 characters | Time of high Humidex |
| Just confirming that the next 2 fields were included by mistake in an emergency release (3.6.12), and are not included in current nor any other version, so have labelled them as error. | |||
| 53 (error) | BA | signed decimal | Labelled as Low Humidex, but not used, (appear in 3.6.12, but no other version) |
| 54 (error) | BB | 5 characters | Labelled as Time of low Humidex, but not used, (appear in 3.6.12, but no other version) |
Example of the file
An extract of a few lines of the dayfile.txt
01/08/11,19.3,61,10:22,12.5,06:58,23.8,14:49,1014.26,20:46,1018.83,09:28,0.0,00:00,0.0,17.8,21.6,4.6,10:44,36,14:14,86,01:56,3.56,8.9,23.8,14:49,23.1,14:50,12.3,06:59,0.0,00:00,12.5,06:58,11.3,00:16,6.9,14:34,354,2.0,1.5 02/08/11,16.1,20,16:55,14.7,06:45,24.2,13:54,1013.79,19:13,1015.65,11:14,0.0,00:00,0.0,18.9,13.7,8.0,15:55,42,20:42,85,06:50,2.79,4.9,24.2,13:54,24.3,13:55,15.1,06:40,0.0,00:00,14.7,06:45,14.8,11:59,7.0,21:09,57,1.0,1.7 03/08/11,14.5,36,17:23,14.9,05:50,24.6,14:46,1012.70,18:44,1015.99,08:34,0.0,00:00,0.0,19.4,17.2,4.8,16:04,50,14:38,79,07:04,3.05,5.8,24.6,14:46,25.4,14:47,15.0,05:50,0.0,00:00,14.9,05:50,14.2,20:01,8.9,00:16,32,0.8,1.9 04/08/11,17.7,16,15:43,14.1,06:20,25.3,15:06,1013.08,18:42,1015.31,08:28,0.0,00:00,0.0,20.2,19.4,8.1,14:12,52,18:20,92,06:55,3.30,9.1,25.3,15:06,26.8,14:55,14.9,06:20,0.0,00:00,14.1,06:20,15.8,14:55,12.5,06:25,36,1.0,2.9 05/08/11,16.1,32,12:52,14.2,06:12,22.2,14:07,1013.89,00:01,1016.36,09:43,0.0,00:00,0.0,18.6,21.6,5.2,13:00,62,15:57,87,06:11,3.30,8.4,22.2,14:07,23.5,14:10,14.8,07:19,0.0,00:00,14.2,06:12,15.4,10:33,12.0,06:03,34,0.9,1.3 06/08/11,16.1,309,11:15,14.3,05:29,22.4,17:12,1014.46,20:02,1016.97,10:38,0.0,00:00,0.0,18.4,19.2,5.5,16:21,55,13:33,92,05:20,2.79,7.9,22.4,17:12,23.3,18:17,15.1,06:09,0.0,00:00,14.3,05:29,14.2,18:12,10.9,10:38,32,1.1,1.3 07/08/11,17.7,342,13:24,12.9,05:47,24.1,14:53,1013.92,19:49,1016.43,09:36,0.0,00:00,0.0,18.4,19.1,6.3,14:06,48,12:45,89,05:36,3.30,9.0,24.1,14:53,24.6,15:48,13.3,05:47,0.0,00:00,12.9,05:47,14.6,15:52,10.7,11:33,11,1.6,1.7