Dayfile.txt: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search
m (Added section 'Viewing the daily log')
m (→‎Introduction: add sentence about Cumulus 1 and mX)
(47 intermediate revisions by 3 users not shown)
Line 1: Line 1:
= Introduction =
Cumulus maintains a daily log file that holds the highs and lows of each day, as well as a few other nuggets of information.  The figures contained in the file are used for the 'This period' display accessed from the '''View''' menu and to build any graphs based on daily values.
Cumulus maintains a daily log file that holds the highs and lows of each day, as well as a few other nuggets of information.  The figures contained in the file are used for the 'This period' display accessed from the '''View''' menu and to build any graphs based on daily values.


== Viewing the daily log ==
The format of this file is the same for both Cumulus 1 and Cumulus MX. It can be ported between them only if both run with exactly the same locale settings, as using a different locale may change the field separator or the symbol used for decimal points.
To view the daily log use the ''dayfile.txt'' command in '''Edit''' menu.  This is how you view the dayfile.txt from within Cumulus. If you upload the file to your web site, then see [[AnnualDataSummary]] for information about ways to show values for a particular weather criterion in a calendar like display.


'''Note:''' there is no option in the '''View''' menu. (The ''data log'' option in that menu is for viewing the [[Monthly_log_files]] only).
== When Cumulus is left running ==
== Dealing with rogue measurements ==
* Cumulus is frequently reading observations from your weather station, but these don't affect the daily log "dayfile.txt" as it is only updated once a day.
Cumulus provides via '''Configuration''' menu ''Calibration'' screen the ability to screen out spikes in data picked up from your weather station. See Cumulus help screen if you decide to use that to cope with some spikes. But some rogue values may be within the accepted all-time range, but just not really applicable to that particular day.
* There are no updates to dayfile.txt at any other times, but the contents of the file are read and processed for many of the display and edit menu options when selected from the main Cumulus 1 screen.
* Cumulus tracks the highs and lows in weather observations by comparing read values against those it has stored in [[Today.ini]], updating that file as required. (It also updates [[Alltime.ini]], [[Monthlyalltime.ini]], [[Year.ini]], and [[Month.ini]] when appropriate.
* Cumulus will not mind you accessing the daily log outside its software, except when it needs write access for processing end of day.
* If you do need to correct some rogue data in the log file, first take a copy and work on that copy, because any edits you do could muck up the specific format that Cumulus 1 or MX needs, there is a section on dealing with rogue data below. Only when you are absolutely sure that your edited copy meets all the constraints listed later, should you replace the original.


If you discover a rogue measurement (perhaps the wind affected your tiping bucket rain gauge or your weather station just reported a corrupted value), on the day it occured, see [[today.ini]] or [[FAQ]] for further advice.
=== When Cumulus processes the end of the (meteorological) day ===
*It updates the daily log, the highs, lows and other information in 'today.ini' are used to append a new line into dayfile.txt, and some of this information is also stored in [[yesterday.ini]].
*Back ups of all the log files mentioned here (not all the log files) are copied to the 'cumulus\backup\daily' folder, a total of 9 sub-folders are retained.
* Some people take a copy of the local file, after it has been updated and store it on (or file transfer it to) their web server. One way of doing this is [[Upload_Dayfile| described here]].
=== When Cumulus is restarted after a break in running  ===
* It reads the daily log and uses the rainfall totals for each day stored in the daily summary log to calculate the rainfall for this month, and this year/season (see [[FAQ#Where_does_Cumulus_get_its_this_month_and_this_year_rainfall_totals_from.3F|FAQ]])
* Thus you must not have another process attempting access to the daily log when Cumulus is re-starting.
* Back ups of all the log files mentioned here (not all the log files) including dayfile.txt are copied to the 'cumulus\backup' folder, the last 8 only are retained.
== How you can use the daily log ==
* If you want to run scripts that use the daily log, it is best if you take a copy first, you can ask Cumulus 1 to take a copy after each update by using the '''Daily''' box in the bottom left of the ''Sites/Options'' frame within the ''Internet'' options screen from the '''Configuration''' menu; that will safely take a copy of 'dayfile.txt' after it is updated. This has the advantage it happens even if Cumulus has been stopped and restarted and rollover is happening during catch-up and so not at usual rollover time according to the computer clock.  See Cumulus 1 '''Help''' for information on using this feature, I add a redirection ">daily_batch.log" in the parameter box alongside so that any output from running the command file I specify in the main box is sent to a log file overwritten in each run; this enables me to see the reason for any failure. 
* Cumulus MX has option to list files to be transferred once a day as part of rollover, so you can use that to generate your extra copy. This has the advantage it happens even if Cumulus has been stopped and restarted and rollover is happening during catch-up.
* A third party toll "Cumulus Toolbox" can also be used to copy/transfer files at a particular time. Note this cannot tell whether Cumulus has done its rollover at the normal time, or during catch-up.
* There are other ways to specify that when a file changes it is copied somewhere.
*The system routines that Cumulus uses to access dayfile.txt require exclusive use of that file, so if you have any other process trying to access that file when Cumulus restarts, when Cumulus processes end of the (meteorological) day, or when a relevant option is selected from View or Edit menus, either your external process or the Cumulus process may fail.


If the rogue measurement is discovered some days after it occured, then in many cases it will have affected your highs and lows for the current month, month-by-month, current year, and/or all-time. As first step you should update the appropriate field in the row for the affected date in dayfile.txt. See [[Alltimelog.txt]] for current and previous values to help you know what rogue value to hunt for and know what the high/low value was before the rogue affected it. Once ''dayfile.txt is correct'' the Cumulus editors will allow you to:
== Populating a database table ==
* update the highs and lows in [[Alltime.ini]] by choosing ''all time records'' from the '''Edit''' menu.
* The [[ImportCumulusFile|article here]] describes a method that can be used with Cumulus 1 to mimic the contents of dayfile.txt in a database table.
* update the highs and lows in [[year.ini]] by choosing ''This year's records'' from the '''Edit''' menu.
* Cumulus MX includes the ability to generate SQL to update a database table version of the log.
* update the highs and lows in [[month.ini]] by choosing ''This month's records'' from the '''Edit''' menu. See [[Diags]] for current and previous values of high or low in the current month or the immediate preceding month if the rogue was recorded less than 10 days ago.
 
* update the highs and lows month-by-month in [[monthlyalltime.ini]] by choosing ''Monthly records'' from the '''Edit''' menu. Click the ''Help'' button for specific instructions on using ''Reset'' and the two ''Copy'' column header buttons in this ''Monthly Records (Highs and Lows) Editor'' to action all rows.
In both cases, your web site can use that database table avoiding any clash of timing with the Cumulus 1 or MX use of the daily summary log. For examples of some of the third party tools using the database daily summary table see [[Daily Summary#Some_example_Scripts|here]].
  '' '''Note''' in each of above 4 editing screens you can:
 
# see the currently stored extremes, and optionally ''Reset'' (row by row) to pre-editing value and timestamp.
Of course you do not need to exactly mimic the log file with the schema in your database table, your weather station may not produce solar values so those fields in dayfile.txt need not be columns in your database table, or you may wish to add other values from external sensors or other log files. MX allows you to specify a different schema in the SQL it generates. In my own case, my daily summary table has no solar columns but it does have several additional columns (including the daily increment of chill hours, the cumulative chill hours, the time of the last rain tip, wind bearings as compass characters (e.g. NNW) as well as numerical bearings). I use Cumulus 1 so I have written the PHP script to find all these additional values, for example it reads the [[today.ini]] stored in the end of day backup, but other sources are also used. In my case I also store the equivalent of what appears on my version of "thismonth.htm" each month in another database table. This monthly summary table allows me to have web pages that compare consecutive months or compare months between years. Again MX has the ability to store end of month figures, a feature that Cumulus 1 and 2 lacked.
# load the dayfile.txt to view extremes derived from those figures (after your correction of the rogue values) and
 
# optionally ''Copy'' (row by row) the logged values (and associated date/time information) into the relevant .ini file.
== Viewing or Editing the daily summary log ==
# click the ''Help'' button for detailed instructions on using ''The Records (Highs and Lows) Editors''.
 
# store your revised figures by clicking ''OK'' (or abandon all your edits by clicking ''Cancel'').
=== Editing the file or other Manipulation outside Cumulus ===
  (Each of these screens is a text editor, and works best when at full screen).''
The file is <tt>data\dayfile.txt</tt> within the directory holding the Cumulus executable, it can be viewed in a text editor, imported into various database systems, or imported into spreadsheets, to manipulate as you wish.  Just remember that Cumulus reads it when it is restarted, and updates it as part of the rollover process, so never attempt to work on it either when Cumulus has just been restarted and is checking/updating (and possibly doing a rollover of logs), or around the midnight/9am/10am local rollover time when Cumulus is writing a new row.
 
'''Tip''':  Take a copy of the file before you do any edit outside Cumulus, so you can revert to old file.
 
'''Note''': Since new versions/builds can add to number of fields, Cumulus will accept lines of 15 fields or more (including without the more recent fields at the end).  (Additions by versions are indicated below, you can explore details of earlier versions via the official [Software software/download] page).
 
'''Important Rules''':
These notes were written for Cumulus 1, at the time this article was updated MX had not introduced any changes to the Cumulus 1 version of this daily log file, but check release notes in case that is no longer true in the future.
* The file must never be edited with a word processor, as they store many control and identification characters that prevent Cumulus correctly reading the values.
* It is recommended that you use either a specialised "Comma Separated Value" file editor or a text editor, both of these can be easily used. You can use a spreadsheet tool, but if you do, there may be a number of settings to change from their defaults to ensure the file remains in a readable format for Cumulus.
* All rows must ''start with date'' and include some of the parameters listed ''in correct sequence''.
* The file should be saved without "Byte Order Mark", specialised text editors will include a menu where you select the encoding and can select not to include BOM.
* The (meteorological) date format uses ''two digits for the year''. This is one reason why you need to edit this file using an editor that treats all fields as text (a text editor, a CSV editor, or a spreadsheet program that can be instructed ''not'' to recognise special field types).  For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to change it to four figure years, and then Cumulus will no longer be able to use the log file.
* (Remember the month must be the middle figure in the date, USA convention cannot apply within logfiles). The separator between the three parts of the date should be a '-' hyphen or a '/' slash, it cannot be a space. Although, use of comma or point for separating parts of the date is allowed by Cumulus, it is not recommended as it can cause issues for subsequent edits.
** If you move your software to a new device, or you change from Cumulus 1 to Cumulus MX (or back), then you must ensure your dates still use the same separator, so all lines are consistent. Equally if you use any third-party packages like for example "Cumulusutils", the separator used in first line is assumed to be true for all lines. Some third-party tools have to be told what separator you use for dates.
* The fields are separated using the Windows (or whatever operating system you are using for MX) list separator (e.g. a comma or semi-colon)[[File:Open office (editing cumulus log files).png]] If you wish to use Excel, or to use "Calc" in 'Apache Open Office', "Libre Office", or similar, you may on opening the file need to pre-select the field separator you use (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" (or whatever similar feature name your tool uses) unselected. Again third party packages processing dayfile.txt will need to recognise your field separator, and some may need to specify it.
* Rows can vary in length but only by missing off ''fields at the end''.
* Most value fields are in ''real number format x.y'' using your system decimal notation, a few (e.g. bearings, solar, humidity) are ''integers'' (see [[#List_of_fields_in_the_file]]). Whilst an integer can be used for a real number field, decimals are not allowed in an integer field.
* If you insert a ''lowest or highest value'' for a new day, where there was no record before, insert a ''time-stamp'' too, as a dayfile.txt row is only accepted by the Cumulus editor if each value has any related time-stamp. (Use a time-stamp of your rollovertime 00:00, 09:00 or 10:00 if you have not looked up the precise time). If you are using a 9am or 10am rollover time, create missing in Cumulus inserts 00:00 for null time-stamps, but normally Cumulus uses the rollover time for null time-stamps.
* Times appearing for some of the fields must always be in ''format HH:mm'' i.e. 2 digit hour, followed by a colon, then 2 digit minutes (Be aware you will have problems if you, or your editing software, add seconds). Except for wind gust (start of line), each time field will immediately follow the value for that parameter.
* Shorter lines can have multiple field separators added at end of row added either when editing within Cumulus or when editing using a spreadsheet tool. (during editing after all available valid parameters inserted, extra field separators may be added at end of shorter lines inserted by 'Create Missing' or by a spreadsheet as these make all lines end up with same number of fields)
* Nulls (2 field separators without something between them ',,') are thus allowed at end of line, but are not allowed within the part of the line with values and time-stamps. If you are editing out rogue values and if you do not know the value for a particular field within the line, then type in a zero or 9999 for nulls in integer format and an extreme with opposite value (e.g. -999.9 for a signed decimal maximum, and 9999.9 for a decimal minimum) for nulls in decimal format (replace the full stops with your decimal separator).
**Beware - if you do insert zero or an obviously wrong extreme value, Cumulus will display those in any editing screen where you wish to update the all-time, monthly-all-time, this month, or this year, extremes.  This can make editing by picking values in logs harder.
**Cumulus itself will use zero for any parameters (e.g. solar) not provided by your station, and will repeat the last valid value if the station fails to send a value it should provide, so if a station fails to send a value for more than a day, dayfile.txt may show the same value as the previous day.
*** Note that Cumulus will stop if your station fails to send what it considers as a vital reading, like pressure or temperature, so the previous point does not apply in all cases.
**Some third-party scripts read the file to calculate averages or other statistics, and their authors suggest you remove rogue values (creating the ',,' that Cumulus objects to). You must never do this in the dayfile.txt that Cumulus processes. My suggestion is use the 'External Program' facility to create a copy of ''dayfile.txt'', make any such changes only on that copy, and set the third-party script to read this copy. 
*** Alternatively, do as I do, clean your data as you upload the file into a database. I use a script with validation code checking for  rogue values and recognising the -999.9 etc., in all cases my script will store a '''NULL''' value as default in the database table if that validation finds any obviously invalid figure.
* The row terminator for Windows is ''CR LF'', ensure any external editor does not change the two character terminator into a single character. Similar rules apply for terminators used by other operating systems, so be careful if you edit the file on a different device to that running Cumulus.
* Make sure that any editing does not create any ''blank lines'' in the file. Cumulus assumes an empty line means end of processing.
* Don't add a header line to the file, Cumulus expects all lines to be data lines.


Ideally, you will subsequently try to edit the rogue data for the particular time it was logged see [[Monthly_log_files#Correcting_any_logged_data_problems]], but correcting the daily summary in dayfile.txt must always be the priority.
=== Editing daily summary in Cumulus ===
'''This section applies to Cumulus 1.x.y only - Cumulus MX v3.0.0 (checked at build 3043) does not provide an editor'''


==Editing in Cumulus==
The last command in '''Edit''' [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu |menu]] is ''dayfile.txt''.  ''This is how you view'' the dayfile.txt from within Cumulus.    This is a text editor, so you can type new values over those currently displayed, insert and delete rows, and it works best when at full screen.
The last command in '''Edit''' menu is ''dayfile.txt''.  This is a text editor, and works best when at full screen.
Click the ''Help'' button for detailed instructions.  Cumulus Help is comprehensive.
Click the ''Help'' button for detailed instructions.  Cumulus Help is comprehensive.
You can use this editor to:
 
*correct individual values by overtyping,
If a particular day does not exist as a row on the daily summary log, then 'create missing' can search the observations in the relevant monthly log, and calculate approximate highs, lows and totals to insert as an exta row in the daily summary log. These are approximate because the actual highs and lows for that day are quite likely to have occurred at moments in-between those that were logged.
 
If there is an error in ''dayfile.txt'', then it is most likely to be found when you are viewing its data on one of the screens for editing the monthly, annual or all-time extremes.  Cumulus will illuminate its ''Error'' light if it finds an error in such cases and tell you the line/row number of the first found error, together with some details of the error it found. For example, if a row is blank, a row is duplicated, a field is corrupted, a field does not have an acceptable value, or a field is missing so subsequent fields are to the left of where they should be.
 
You can use this editor as follows:
*use '''insert''' key to add one or more missing rows (complete days) manually,
*correct individual values by over-typing,
*use '''delete''' key to remove an entire day (e.g. if you get a 'duplicate' error message),
*use '''delete''' key to remove an entire day (e.g. if you get a 'duplicate' error message),
*use '''insert''' key to add one or more missing rows (complete days) manually,
*use '''Create missing''' button to insert missing rows (complete days) by reading from [[Monthly_log_files]] and automatically calculating the best approximations for each field for those missing days.
*use '''Create missing''' button to insert missing rows (complete days) by reading from [[Monthly_log_files]] and automatically calculating the best approximations for each field for those missing days.
If you do have a 'duplicate' error, you need to decide which row to ''delete'', and whether to copy any values from that row into the row you are keeping to ensure the correct extremes are retained.
If you do have a 'duplicate' error, you need to decide which row to ''delete'', and whether to copy any values from that row into the row you are keeping to ensure the correct extremes are retained.
As an alternative to manual line  ''insert'', you may wish to use a procedure for importing, and processing, pre-Cumulus observations into [[Monthly_log_files]].  Once there is data for required days in monthly logs, ''Create missing'' can insert the new rows for those days previously missing in dayfile.txt.


For ''Create missing'' a list of inserted records is produced in [[dayfileeditlog.txt]]. If just some fields are wrong in a particular row (meteorological day) on day file, then there is a [[Monthly_log_files#Using_Monthly_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] as at all current versions (up to 1.9.4) you can only use 'Create missing' to read from the [[Monthly_log_files]] if the whole day (a row starting with a date) is missing in ''dayfile.txt''.
For ''Create missing'' a list of inserted records is produced in [[dayfileeditlog.txt]]. If just some fields are wrong in a particular row (meteorological day) on day file, then there is a [[Monthly_log_files#Using_Monthly_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] as at all current versions (up to 1.9.4) you can only use 'Create missing' to read from the [[Monthly_log_files]] if the whole day (a row starting with a date) is missing in ''dayfile.txt''.
=== Read this if you are not using the latest release of Cumulus ===


The procedure for importing, and processing, pre-Cumulus observations is explained in [[Monthly_log_files |this Wiki topic]].  
'''Note for obsolete version 1.9.0 to 1.9.3:''' There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.


'''Note for version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause incorrect date order for records (dayfile.txt uses dd/mm/yy  or dd-mm-yy and all records should be in ascending chronological order)
'''Note for obsolete version 1.9.3 only:''' Create missing might in some cases be affected by a bug in 1.9.3 that can cause incorrect date order for records (dayfile.txt uses dd/mm/yy  or dd-mm-yy and all records should be in ascending chronological order)


'''Note for versions 1.9.2 or version 1.9.3''' There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.
There are no known bugs in version 1.9.4 builds 1086 to 1100. Build 1099 is the standard release as this section was updated.
=== Dealing with rogue measurements ===
Cumulus provides via '''Configuration''' menu ''Calibration'' screen the ability to screen out spikes (magnitude of differences between one value read and next value read) in data picked up from your weather station. See Cumulus help screen if you decide to use that to cope with some spikes.


Any bugs are believed fixed in version 1.9.4.
If you discover a rogue measurement (perhaps the wind affected your tipping bucket rain gauge or your weather station just reported a corrupted value), on the day it occured, see [[today.ini]] or a [[FAQ]] for further advice. In general, you need to stop Cumulus, edit the monthly log row containing the dodgy values, edit 'today.ini' and possibly other '.ini' files, looking up the logs indicated below that show the updates with previous high or low.


==Manipulation outside Cumulus==
If the rogue measurement is discovered some days after it occured, then in many cases it will have affected your highs and lows for the current month, month-by-month, current year, and/or all-time.  As your first step you should update the appropriate field in the row for the affected date in dayfile.txt. Once ''dayfile.txt is correct'' the Cumulus editors will allow you to:
The file is called <tt>dayfile.txt</tt> which can be viewed in a text editor or imported into various database systems or spreadsheets to manipulate as you wishJust remember that Cumulus updates it as part of the rollover process, so never attempt to work on it either when Cumulus has just been restarted and is checking/updating the logs, or around the midnight/9am/10am local rollover time when Cumulus is writing a new row.
* update the highs and lows in [[Alltime.ini]] by choosing ''all time records'' from the '''Edit''' menu.  See [[Alltimelog.txt]] for current and previous values to help you know what rogue value to hunt for and know what the high/low value was before the rogue affected it.
* update the highs and lows in [[year.ini]] by choosing ''This year's records'' from the '''Edit''' menu.
* update the highs and lows in [[month.ini]] by choosing ''This month's records'' from the '''Edit''' menu. See [[Diags]] for current and previous values of high or low in the current month or the immediate preceding month if the rogue was recorded less than 10 days ago.
* update the highs and lows month-by-month in [[monthlyalltime.ini]] by choosing ''Monthly records'' from the '''Edit''' menu. Click the ''Help'' button for specific instructions on using ''Reset'' and the two ''Copy'' column header buttons in this ''Monthly Records (Highs and Lows) Editor'' to action all rows.
* '' '''Note''' in each of above 4 editing screens you can:
*# see the currently stored extremes, and optionally ''Reset'' (row by row) to pre-editing value and timestamp.
*# load the dayfile.txt to view extremes derived from those figures (after your correction of the rogue values) and
*# optionally ''Copy'' (row by row) the logged values (and associated date/time information) into the relevant .ini file.
*# click the ''Help'' button for detailed instructions on using ''The Records (Highs and Lows) Editors''.
*# store your revised figures by clicking ''OK'' (or abandon all your edits by clicking ''Cancel'').
  (Each of these screens is a text editor, and works best when at full screen).''
* create the relevant monthly and/or annual NOAA style report by choosing NOAA Monthly Report or NOAA Annual Report from the View menu, then select the required period using the selectors. Click the Update Display button to see various statistics (including mean temperature) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from the monthly log files) for period in question. Finally press Save button to store the new or amended report.
Ideally, you will subsequently try to edit the rogue data for the particular time it was logged; see [[Monthly_log_files#Correcting_any_logged_data_problems]], but correcting the daily summary in dayfile.txt must always be the priority.
 
== Using the daily summary log on your web-site ==


'''Tip''':  Take a copy of the file before you work on it outside Cumulus (perhaps call the copy 'dayfile.csv' so Windows can recognise its structure).
If you upload the log file to your web site then (with the help of JavaScript) you can read the log file to obtain information to show on a web page.  You could have a web page that shows a today.htm like table for the last 7 days by combining reading Cumulus web tags with reading from the log file.  


'''Note''': Since new versions/builds can add to number of fields, Cumulus will accept lines of various lengths without the more recent fields at the end. (Additions for the last few builds only are indicated below, you can explore details of earlier versions via the official [http://sandaysoft.com/products/cumulus product] page).
Search the Cumulus support forum to see (for example) how others extract information from dayfile.txt to display on their web page a set of fields similar to those shown for 'Yesterday.htm' web page for other dates in the past, such as one year ago.
 
If you use a script to read what is in the  daily summary log file into a database table, or use the functionality in Cumulus MX to upload automatically to a database table, then see [[Daily_Summary]] article for information about ALL of the ways to show values from this database table.
 
==Viewing summary figures for a month or period==
To view a summary of dayfile.txt for a month, calendar year or selected period, use ''This month'' (choose any month, default is month from your computer system date), ''This year'' (choose any year, default is year from your computer system date), or ''This period'' (choose any start and end dates, default is yesterday calculated from your computer system date), within the '''View''' [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|menu]].
*Remember the daily summary log has its records based on rollover to rollover days.
*In all cases they exclude the today details that are not stored on dayfile.txt until the end of day rollover.
*If you use 9am or 10am rollover, and choose '''View''' ''This period'' between midnight and your 9am/10am rollover any day your latest meteorological day is the yesterday in terms of your computer system date that 'This period' tries to display as its default day, and the display will initially appear blank.
*If you use 9am or 10am rollover, and choose '''View''' ''This month'' before your 9am/10am rollover on the first day of a new calendar month your latest meteorological month is different to your computer system month 'This month' tries to display as its default month, and the display will initially appear blank.
 
Most of the displayed results are for observations in the daily summary log, but a few parameters are not in that log and are derived from the monthly logs (e.g. average wind speed) or the weather diary (e.g. count of days with snow lying).
*On the screen displayed after selecting ''This month'', you can change the month and year required using the options at bottom left, click ''Update Display'' and the revised summary will be calculated.
*On the screen displayed after selecting ''This year'', you can change the year required using the options at bottom left, click ''Update Display'' and the revised summary will be calculated.
*On the screen displayed after selecting ''This period'', you can change the start date and end date then click ''Update Display'' to get the equivalent calculations displayed for part of a month or any other period.
 
 
Note differences between observation reports on View screens and those available as web tags.
*Date and time stamps:
**The day number shown on screen is the meteorological day (changing at rollover and that may be at midnight or 9am/10am) as that date appears in dayfile.txt;
**A time-stamp (with time and date) given in a web tag quotes a calendar date (always changing at midnight).
*Reported statistics example: 
**The screen shows total number of dry or wet days in the month;
**The web tags report longest dry or wet period in the month.


'''Important Rules''':
* The (meteorological) date format uses ''two digits for the year''. Edit this file using an editor that treats all fields as text (a text editor or a spreadsheet program that can be instructed ''not'' to recognise special field types).  -  ''Do not edit this file using Excel'' or any similar tool that recognizes dates, as it is likely to change the dates to have four figure years, and then Cumulus will no longer be able to use the file.
* Times are in ''format hh:mm'', and most value fields are in format x.y using your system decimal notation, a few (e.g. bearings, solar, humidity) are integers.
* If you insert a ''lowest or highest value'' into a day, where there was none before, insert a ''timestamp'' too, as a dayfile.txt row is only accepted by the Cumulus editor if a value and any related timestamp are either both present or both absent. (Use a timestamp of your rollovertime 00:00, 09:00 or 10:00 if you have not looked up the precise time).
* Nulls ',,' are not allowed in the line, so if you do not know the value for a particular field within the line, then type in a zero for integer values and an extreme with opposite value (e.g. -999.9 for a signed decimal maximum, and 999.9 for a decimal minimum) for decimal format (replace the full stops with your decimal separator).
* Make sure that any editing does not create any ''blank lines'' in the file.
* All rows must ''start with date'' and include some of the parameters listed ''in correct sequence''.
* Rows can vary in length but only by missing off ''fields at the end''.
* The row terminator normally expected is ''CRLF'', ensure any external editor does not change the terminator.


==List of fields in the file==
==List of fields in the file==
*Alternative lists:
'''The appropriate list of fields for your installed build is stored as ''dayfileheader.txt'' within the folder that contains your Cumulus executable.'''
**A version of this list with spreadsheet column letters can be downloaded from the support forum  [http://sandaysoft.com/forum/viewtopic.php?f=18&t=8690 here],
**the appropriate list of fields for your installed build is stored as dayfileheader.txt within the folder that contains your Cumulus executable.
 
The field number (starting from zero to be consistent with index used for arrays in programming languages like JavaScript) with description and cross-references to further explanation:
* 00: Date as  ''2 figure day [separator] 2 figure month [separator] 2 figure year'' - the separator is that set in the windows system short date format (see [[setup]])
* 01: Highest wind gust
* 02: [[Wind_measurement#Wind_Direction | Bearing]] of highest wind gust (integer)
* 03: Time of highest wind gust
* 04: Minimum [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | temperature]]
* 05: Time of minimum temperature
* 06: Maximum temperature
* 07: Time of maximum temperature
* 08: Minimum [[Pressure_Measurement | sea level pressure]]
* 09: Time of minimum pressure
* 10: Maximum sea level pressure
* 11: Time of maximum pressure
* 12: Maximum [[Rain_measurement#Rain_Rate | rainfall rate]]
* 13: Time of maximum rainfall rate
* 14: Total rainfall for the day
* 15: [[Average temperature]] for the day
* 16: Total [[Windrun | wind run]]
* 17: Highest Average Wind Speed
* 18: Time of Highest Avg. Wind speed
* 19: Lowest [[Temperature_(and_humidity)_measurement | humidity]]
* 20: Time of lowest humidity
* 21: Highest humidity
* 22: Time of highest humidity
* 23: Total evapotranspiration
* 24: Total hours of sunshine
* 25: High [[Heat index]]
* 26: Time of high heat index
* 27: High [[Apparent temperature]]
* 28: Time of high apparent temperature
* 29: Low apparent temperature
* 30: Time of low apparent temperature
* 31: High hourly rain
* 32: Time of high hourly rain
* 33: Low [[wind chill]]
* 34: Time of low wind chill
* 35: High [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | dew point]]
* 36: Time of high dew point
* 37: Low dew point
* 38: Time of low dew point


The next 3 entries were added in version 1.9.2 Build 1004
Listed below is the field number (starting from zero to be consistent with index used for arrays in programming languages like JavaScript, plus spreadsheet column identifier) with description and cross-references to further explanation:
* 39: Today's dominant/average wind direction (integer)
* 00 (A): Date as  ''2 figure day [separator] 2 figure month [separator] 2 figure year'' - the separator is that set in the windows system short date format (see [[setup]])
* 40: [[Heat/cold degree days and Chill hours | Heating degree days]]
* 01 (B): Highest wind [[Wind_measurement#Weather_Stations_and_Cumulus|gust]]
* 41: [[Heat/cold degree days and Chill hours | Cooling degree days]]
* 02 (C): [[Wind_measurement#Wind_Direction | Bearing]] of highest wind gust (integer)
* 03 (D): Time of highest wind gust
* 04 (E): Minimum [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | temperature]]
* 05 (F): Time of minimum temperature
* 06 (G): Maximum temperature
* 07 (H): Time of maximum temperature
* 08 (I): Minimum [[Pressure_Measurement | sea level pressure]]
* 09 (J): Time of minimum pressure
* 10 (K): Maximum sea level pressure
* 11 (L): Time of maximum pressure
* 12 (M): Maximum [[Rain_measurement#Rain_Rate | rainfall rate]]
* 13 (N): Time of maximum rainfall rate
* 14 (O): Total rainfall for the day
(This point represents the minimum length of all records)
* 15 (P): [[Average temperature]] for the day
* 16 (Q): Total [[Windrun | wind run]]
(The last two entries were added from v 1.8.9 build 907)
* 17 (R): Highest [[Wind_measurement#Weather_Stations_and_Cumulus|Average Wind Speed]]
* 18 (S): Time of Highest Avg. Wind speed
(Those two entries were added from v 1.9.0)
* 19 (T): Lowest [[Temperature_(and_humidity)_measurement | humidity]] (integer)
* 20 (U): Time of lowest humidity
* 21 (V): Highest humidity (integer)
* 22 (W): Time of highest humidity
* 23 (X): Total evapotranspiration (Only valid for Davis stations, shows zero otherwise)
* 24 (Y): Total hours of sunshine (only valid if sunshine sensor connnected)
* 25 (Z): High [[Heat index]]
* 26 (AA): Time of high heat index
* 27 (AB): High [[Apparent temperature]]
* 28 (AC): Time of high apparent temperature
* 29 (AD): Low apparent temperature
* 30 (AE): Time of low apparent temperature
* 31 (AF): High hourly rain
* 32 (AG): Time of high hourly rain
* 33 (AH): Low [[wind chill]]
* 34 (AI): Time of low wind chill
(The last 16 entries were added in version 1.9.1)
* 35 (AJ): High [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | dew point]]
* 36 (AK): Time of high dew point
* 37 (AL): Low dew point
* 38 (AM): Time of low dew point
(Last 4 entries added in version 1.9.2 beta build)
* 39 (AN): Today's dominant/average wind direction (integer)
* 40 (AO): [[Heat/cold degree days and Chill hours | Heating degree days]]
* 41 (AP): [[Heat/cold degree days and Chill hours | Cooling degree days]]
(These last three entries were added from 1.9.2 Build 1004)


Added in version 1.9.3 build 1036
Finally, added in version 1.9.3 build 1036 (these only show valid values if appropriate sensors exist)
* 42: High solar radiation
* 42 (AQ): High solar radiation
* 43: Time of high solar radiation
* 43 (AR): Time of high solar radiation
* 44: High UV Index
* 44 (AS): High UV Index
* 45: Time of high UV Index
* 45 (AT): Time of high UV Index
(Cumulus MX uses all fields as listed above)


==Example of the file==
==Example of the file==

Revision as of 13:55, 18 March 2020

Introduction

Cumulus maintains a daily log file that holds the highs and lows of each day, as well as a few other nuggets of information. The figures contained in the file are used for the 'This period' display accessed from the View menu and to build any graphs based on daily values.

The format of this file is the same for both Cumulus 1 and Cumulus MX. It can be ported between them only if both run with exactly the same locale settings, as using a different locale may change the field separator or the symbol used for decimal points.

When Cumulus is left running

  • Cumulus is frequently reading observations from your weather station, but these don't affect the daily log "dayfile.txt" as it is only updated once a day.
  • There are no updates to dayfile.txt at any other times, but the contents of the file are read and processed for many of the display and edit menu options when selected from the main Cumulus 1 screen.
  • Cumulus tracks the highs and lows in weather observations by comparing read values against those it has stored in Today.ini, updating that file as required. (It also updates Alltime.ini, Monthlyalltime.ini, Year.ini, and Month.ini when appropriate.
  • Cumulus will not mind you accessing the daily log outside its software, except when it needs write access for processing end of day.
  • If you do need to correct some rogue data in the log file, first take a copy and work on that copy, because any edits you do could muck up the specific format that Cumulus 1 or MX needs, there is a section on dealing with rogue data below. Only when you are absolutely sure that your edited copy meets all the constraints listed later, should you replace the original.

When Cumulus processes the end of the (meteorological) day

  • It updates the daily log, the highs, lows and other information in 'today.ini' are used to append a new line into dayfile.txt, and some of this information is also stored in yesterday.ini.
  • Back ups of all the log files mentioned here (not all the log files) are copied to the 'cumulus\backup\daily' folder, a total of 9 sub-folders are retained.
  • Some people take a copy of the local file, after it has been updated and store it on (or file transfer it to) their web server. One way of doing this is described here.

When Cumulus is restarted after a break in running

  • It reads the daily log and uses the rainfall totals for each day stored in the daily summary log to calculate the rainfall for this month, and this year/season (see FAQ)
  • Thus you must not have another process attempting access to the daily log when Cumulus is re-starting.
  • Back ups of all the log files mentioned here (not all the log files) including dayfile.txt are copied to the 'cumulus\backup' folder, the last 8 only are retained.

How you can use the daily log

  • If you want to run scripts that use the daily log, it is best if you take a copy first, you can ask Cumulus 1 to take a copy after each update by using the Daily box in the bottom left of the Sites/Options frame within the Internet options screen from the Configuration menu; that will safely take a copy of 'dayfile.txt' after it is updated. This has the advantage it happens even if Cumulus has been stopped and restarted and rollover is happening during catch-up and so not at usual rollover time according to the computer clock. See Cumulus 1 Help for information on using this feature, I add a redirection ">daily_batch.log" in the parameter box alongside so that any output from running the command file I specify in the main box is sent to a log file overwritten in each run; this enables me to see the reason for any failure.
  • Cumulus MX has option to list files to be transferred once a day as part of rollover, so you can use that to generate your extra copy. This has the advantage it happens even if Cumulus has been stopped and restarted and rollover is happening during catch-up.
  • A third party toll "Cumulus Toolbox" can also be used to copy/transfer files at a particular time. Note this cannot tell whether Cumulus has done its rollover at the normal time, or during catch-up.
  • There are other ways to specify that when a file changes it is copied somewhere.
  • The system routines that Cumulus uses to access dayfile.txt require exclusive use of that file, so if you have any other process trying to access that file when Cumulus restarts, when Cumulus processes end of the (meteorological) day, or when a relevant option is selected from View or Edit menus, either your external process or the Cumulus process may fail.

Populating a database table

  • The article here describes a method that can be used with Cumulus 1 to mimic the contents of dayfile.txt in a database table.
  • Cumulus MX includes the ability to generate SQL to update a database table version of the log.

In both cases, your web site can use that database table avoiding any clash of timing with the Cumulus 1 or MX use of the daily summary log. For examples of some of the third party tools using the database daily summary table see here.

Of course you do not need to exactly mimic the log file with the schema in your database table, your weather station may not produce solar values so those fields in dayfile.txt need not be columns in your database table, or you may wish to add other values from external sensors or other log files. MX allows you to specify a different schema in the SQL it generates. In my own case, my daily summary table has no solar columns but it does have several additional columns (including the daily increment of chill hours, the cumulative chill hours, the time of the last rain tip, wind bearings as compass characters (e.g. NNW) as well as numerical bearings). I use Cumulus 1 so I have written the PHP script to find all these additional values, for example it reads the today.ini stored in the end of day backup, but other sources are also used. In my case I also store the equivalent of what appears on my version of "thismonth.htm" each month in another database table. This monthly summary table allows me to have web pages that compare consecutive months or compare months between years. Again MX has the ability to store end of month figures, a feature that Cumulus 1 and 2 lacked.

Viewing or Editing the daily summary log

Editing the file or other Manipulation outside Cumulus

The file is data\dayfile.txt within the directory holding the Cumulus executable, it can be viewed in a text editor, imported into various database systems, or imported into spreadsheets, to manipulate as you wish. Just remember that Cumulus reads it when it is restarted, and updates it as part of the rollover process, so never attempt to work on it either when Cumulus has just been restarted and is checking/updating (and possibly doing a rollover of logs), or around the midnight/9am/10am local rollover time when Cumulus is writing a new row.

Tip: Take a copy of the file before you do any edit outside Cumulus, so you can revert to old file.

Note: Since new versions/builds can add to number of fields, Cumulus will accept lines of 15 fields or more (including without the more recent fields at the end). (Additions by versions are indicated below, you can explore details of earlier versions via the official [Software software/download] page).

Important Rules: These notes were written for Cumulus 1, at the time this article was updated MX had not introduced any changes to the Cumulus 1 version of this daily log file, but check release notes in case that is no longer true in the future.

  • The file must never be edited with a word processor, as they store many control and identification characters that prevent Cumulus correctly reading the values.
  • It is recommended that you use either a specialised "Comma Separated Value" file editor or a text editor, both of these can be easily used. You can use a spreadsheet tool, but if you do, there may be a number of settings to change from their defaults to ensure the file remains in a readable format for Cumulus.
  • All rows must start with date and include some of the parameters listed in correct sequence.
  • The file should be saved without "Byte Order Mark", specialised text editors will include a menu where you select the encoding and can select not to include BOM.
  • The (meteorological) date format uses two digits for the year. This is one reason why you need to edit this file using an editor that treats all fields as text (a text editor, a CSV editor, or a spreadsheet program that can be instructed not to recognise special field types). For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to change it to four figure years, and then Cumulus will no longer be able to use the log file.
  • (Remember the month must be the middle figure in the date, USA convention cannot apply within logfiles). The separator between the three parts of the date should be a '-' hyphen or a '/' slash, it cannot be a space. Although, use of comma or point for separating parts of the date is allowed by Cumulus, it is not recommended as it can cause issues for subsequent edits.
    • If you move your software to a new device, or you change from Cumulus 1 to Cumulus MX (or back), then you must ensure your dates still use the same separator, so all lines are consistent. Equally if you use any third-party packages like for example "Cumulusutils", the separator used in first line is assumed to be true for all lines. Some third-party tools have to be told what separator you use for dates.
  • The fields are separated using the Windows (or whatever operating system you are using for MX) list separator (e.g. a comma or semi-colon)Open office (editing cumulus log files).png If you wish to use Excel, or to use "Calc" in 'Apache Open Office', "Libre Office", or similar, you may on opening the file need to pre-select the field separator you use (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" (or whatever similar feature name your tool uses) unselected. Again third party packages processing dayfile.txt will need to recognise your field separator, and some may need to specify it.
  • Rows can vary in length but only by missing off fields at the end.
  • Most value fields are in real number format x.y using your system decimal notation, a few (e.g. bearings, solar, humidity) are integers (see #List_of_fields_in_the_file). Whilst an integer can be used for a real number field, decimals are not allowed in an integer field.
  • If you insert a lowest or highest value for a new day, where there was no record before, insert a time-stamp too, as a dayfile.txt row is only accepted by the Cumulus editor if each value has any related time-stamp. (Use a time-stamp of your rollovertime 00:00, 09:00 or 10:00 if you have not looked up the precise time). If you are using a 9am or 10am rollover time, create missing in Cumulus inserts 00:00 for null time-stamps, but normally Cumulus uses the rollover time for null time-stamps.
  • Times appearing for some of the fields must always be in format HH:mm i.e. 2 digit hour, followed by a colon, then 2 digit minutes (Be aware you will have problems if you, or your editing software, add seconds). Except for wind gust (start of line), each time field will immediately follow the value for that parameter.
  • Shorter lines can have multiple field separators added at end of row added either when editing within Cumulus or when editing using a spreadsheet tool. (during editing after all available valid parameters inserted, extra field separators may be added at end of shorter lines inserted by 'Create Missing' or by a spreadsheet as these make all lines end up with same number of fields)
  • Nulls (2 field separators without something between them ',,') are thus allowed at end of line, but are not allowed within the part of the line with values and time-stamps. If you are editing out rogue values and if you do not know the value for a particular field within the line, then type in a zero or 9999 for nulls in integer format and an extreme with opposite value (e.g. -999.9 for a signed decimal maximum, and 9999.9 for a decimal minimum) for nulls in decimal format (replace the full stops with your decimal separator).
    • Beware - if you do insert zero or an obviously wrong extreme value, Cumulus will display those in any editing screen where you wish to update the all-time, monthly-all-time, this month, or this year, extremes. This can make editing by picking values in logs harder.
    • Cumulus itself will use zero for any parameters (e.g. solar) not provided by your station, and will repeat the last valid value if the station fails to send a value it should provide, so if a station fails to send a value for more than a day, dayfile.txt may show the same value as the previous day.
      • Note that Cumulus will stop if your station fails to send what it considers as a vital reading, like pressure or temperature, so the previous point does not apply in all cases.
    • Some third-party scripts read the file to calculate averages or other statistics, and their authors suggest you remove rogue values (creating the ',,' that Cumulus objects to). You must never do this in the dayfile.txt that Cumulus processes. My suggestion is use the 'External Program' facility to create a copy of dayfile.txt, make any such changes only on that copy, and set the third-party script to read this copy.
      • Alternatively, do as I do, clean your data as you upload the file into a database. I use a script with validation code checking for rogue values and recognising the -999.9 etc., in all cases my script will store a NULL value as default in the database table if that validation finds any obviously invalid figure.
  • The row terminator for Windows is CR LF, ensure any external editor does not change the two character terminator into a single character. Similar rules apply for terminators used by other operating systems, so be careful if you edit the file on a different device to that running Cumulus.
  • Make sure that any editing does not create any blank lines in the file. Cumulus assumes an empty line means end of processing.
  • Don't add a header line to the file, Cumulus expects all lines to be data lines.

Editing daily summary in Cumulus

This section applies to Cumulus 1.x.y only - Cumulus MX v3.0.0 (checked at build 3043) does not provide an editor

The last command in Edit menu is dayfile.txt. This is how you view the dayfile.txt from within Cumulus. This is a text editor, so you can type new values over those currently displayed, insert and delete rows, and it works best when at full screen. Click the Help button for detailed instructions. Cumulus Help is comprehensive.

If a particular day does not exist as a row on the daily summary log, then 'create missing' can search the observations in the relevant monthly log, and calculate approximate highs, lows and totals to insert as an exta row in the daily summary log. These are approximate because the actual highs and lows for that day are quite likely to have occurred at moments in-between those that were logged.

If there is an error in dayfile.txt, then it is most likely to be found when you are viewing its data on one of the screens for editing the monthly, annual or all-time extremes. Cumulus will illuminate its Error light if it finds an error in such cases and tell you the line/row number of the first found error, together with some details of the error it found. For example, if a row is blank, a row is duplicated, a field is corrupted, a field does not have an acceptable value, or a field is missing so subsequent fields are to the left of where they should be.

You can use this editor as follows:

  • use insert key to add one or more missing rows (complete days) manually,
  • correct individual values by over-typing,
  • use delete key to remove an entire day (e.g. if you get a 'duplicate' error message),
  • use Create missing button to insert missing rows (complete days) by reading from Monthly_log_files and automatically calculating the best approximations for each field for those missing days.

If you do have a 'duplicate' error, you need to decide which row to delete, and whether to copy any values from that row into the row you are keeping to ensure the correct extremes are retained.

As an alternative to manual line insert, you may wish to use a procedure for importing, and processing, pre-Cumulus observations into Monthly_log_files. Once there is data for required days in monthly logs, Create missing can insert the new rows for those days previously missing in dayfile.txt.

For Create missing a list of inserted records is produced in dayfileeditlog.txt. If just some fields are wrong in a particular row (meteorological day) on day file, then there is a workaround as at all current versions (up to 1.9.4) you can only use 'Create missing' to read from the Monthly_log_files if the whole day (a row starting with a date) is missing in dayfile.txt.

Read this if you are not using the latest release of Cumulus

Note for obsolete version 1.9.0 to 1.9.3: There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.

Note for obsolete version 1.9.3 only: Create missing might in some cases be affected by a bug in 1.9.3 that can cause incorrect date order for records (dayfile.txt uses dd/mm/yy or dd-mm-yy and all records should be in ascending chronological order)

There are no known bugs in version 1.9.4 builds 1086 to 1100. Build 1099 is the standard release as this section was updated.

Dealing with rogue measurements

Cumulus provides via Configuration menu Calibration screen the ability to screen out spikes (magnitude of differences between one value read and next value read) in data picked up from your weather station. See Cumulus help screen if you decide to use that to cope with some spikes.

If you discover a rogue measurement (perhaps the wind affected your tipping bucket rain gauge or your weather station just reported a corrupted value), on the day it occured, see today.ini or a FAQ for further advice. In general, you need to stop Cumulus, edit the monthly log row containing the dodgy values, edit 'today.ini' and possibly other '.ini' files, looking up the logs indicated below that show the updates with previous high or low.

If the rogue measurement is discovered some days after it occured, then in many cases it will have affected your highs and lows for the current month, month-by-month, current year, and/or all-time. As your first step you should update the appropriate field in the row for the affected date in dayfile.txt. Once dayfile.txt is correct the Cumulus editors will allow you to:

  • update the highs and lows in Alltime.ini by choosing all time records from the Edit menu. See Alltimelog.txt for current and previous values to help you know what rogue value to hunt for and know what the high/low value was before the rogue affected it.
  • update the highs and lows in year.ini by choosing This year's records from the Edit menu.
  • update the highs and lows in month.ini by choosing This month's records from the Edit menu. See Diags for current and previous values of high or low in the current month or the immediate preceding month if the rogue was recorded less than 10 days ago.
  • update the highs and lows month-by-month in monthlyalltime.ini by choosing Monthly records from the Edit menu. Click the Help button for specific instructions on using Reset and the two Copy column header buttons in this Monthly Records (Highs and Lows) Editor to action all rows.
  • Note in each of above 4 editing screens you can:
    1. see the currently stored extremes, and optionally Reset (row by row) to pre-editing value and timestamp.
    2. load the dayfile.txt to view extremes derived from those figures (after your correction of the rogue values) and
    3. optionally Copy (row by row) the logged values (and associated date/time information) into the relevant .ini file.
    4. click the Help button for detailed instructions on using The Records (Highs and Lows) Editors.
    5. store your revised figures by clicking OK (or abandon all your edits by clicking Cancel).
 (Each of these screens is a text editor, and works best when at full screen).
  • create the relevant monthly and/or annual NOAA style report by choosing NOAA Monthly Report or NOAA Annual Report from the View menu, then select the required period using the selectors. Click the Update Display button to see various statistics (including mean temperature) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from the monthly log files) for period in question. Finally press Save button to store the new or amended report.

Ideally, you will subsequently try to edit the rogue data for the particular time it was logged; see Monthly_log_files#Correcting_any_logged_data_problems, but correcting the daily summary in dayfile.txt must always be the priority.

Using the daily summary log on your web-site

If you upload the log file to your web site then (with the help of JavaScript) you can read the log file to obtain information to show on a web page. You could have a web page that shows a today.htm like table for the last 7 days by combining reading Cumulus web tags with reading from the log file.

Search the Cumulus support forum to see (for example) how others extract information from dayfile.txt to display on their web page a set of fields similar to those shown for 'Yesterday.htm' web page for other dates in the past, such as one year ago.

If you use a script to read what is in the daily summary log file into a database table, or use the functionality in Cumulus MX to upload automatically to a database table, then see Daily_Summary article for information about ALL of the ways to show values from this database table.

Viewing summary figures for a month or period

To view a summary of dayfile.txt for a month, calendar year or selected period, use This month (choose any month, default is month from your computer system date), This year (choose any year, default is year from your computer system date), or This period (choose any start and end dates, default is yesterday calculated from your computer system date), within the View menu.

  • Remember the daily summary log has its records based on rollover to rollover days.
  • In all cases they exclude the today details that are not stored on dayfile.txt until the end of day rollover.
  • If you use 9am or 10am rollover, and choose View This period between midnight and your 9am/10am rollover any day your latest meteorological day is the yesterday in terms of your computer system date that 'This period' tries to display as its default day, and the display will initially appear blank.
  • If you use 9am or 10am rollover, and choose View This month before your 9am/10am rollover on the first day of a new calendar month your latest meteorological month is different to your computer system month 'This month' tries to display as its default month, and the display will initially appear blank.

Most of the displayed results are for observations in the daily summary log, but a few parameters are not in that log and are derived from the monthly logs (e.g. average wind speed) or the weather diary (e.g. count of days with snow lying).

  • On the screen displayed after selecting This month, you can change the month and year required using the options at bottom left, click Update Display and the revised summary will be calculated.
  • On the screen displayed after selecting This year, you can change the year required using the options at bottom left, click Update Display and the revised summary will be calculated.
  • On the screen displayed after selecting This period, you can change the start date and end date then click Update Display to get the equivalent calculations displayed for part of a month or any other period.


Note differences between observation reports on View screens and those available as web tags.

  • Date and time stamps:
    • The day number shown on screen is the meteorological day (changing at rollover and that may be at midnight or 9am/10am) as that date appears in dayfile.txt;
    • A time-stamp (with time and date) given in a web tag quotes a calendar date (always changing at midnight).
  • Reported statistics example:
    • The screen shows total number of dry or wet days in the month;
    • The web tags report longest dry or wet period in the month.


List of fields in the file

The appropriate list of fields for your installed build is stored as dayfileheader.txt within the folder that contains your Cumulus executable.

Listed below is the field number (starting from zero to be consistent with index used for arrays in programming languages like JavaScript, plus spreadsheet column identifier) with description and cross-references to further explanation:

  • 00 (A): Date as 2 figure day [separator] 2 figure month [separator] 2 figure year - the separator is that set in the windows system short date format (see setup)
  • 01 (B): Highest wind gust
  • 02 (C): Bearing of highest wind gust (integer)
  • 03 (D): Time of highest wind gust
  • 04 (E): Minimum temperature
  • 05 (F): Time of minimum temperature
  • 06 (G): Maximum temperature
  • 07 (H): Time of maximum temperature
  • 08 (I): Minimum sea level pressure
  • 09 (J): Time of minimum pressure
  • 10 (K): Maximum sea level pressure
  • 11 (L): Time of maximum pressure
  • 12 (M): Maximum rainfall rate
  • 13 (N): Time of maximum rainfall rate
  • 14 (O): Total rainfall for the day

(This point represents the minimum length of all records)

(The last two entries were added from v 1.8.9 build 907)

(Those two entries were added from v 1.9.0)

  • 19 (T): Lowest humidity (integer)
  • 20 (U): Time of lowest humidity
  • 21 (V): Highest humidity (integer)
  • 22 (W): Time of highest humidity
  • 23 (X): Total evapotranspiration (Only valid for Davis stations, shows zero otherwise)
  • 24 (Y): Total hours of sunshine (only valid if sunshine sensor connnected)
  • 25 (Z): High Heat index
  • 26 (AA): Time of high heat index
  • 27 (AB): High Apparent temperature
  • 28 (AC): Time of high apparent temperature
  • 29 (AD): Low apparent temperature
  • 30 (AE): Time of low apparent temperature
  • 31 (AF): High hourly rain
  • 32 (AG): Time of high hourly rain
  • 33 (AH): Low wind chill
  • 34 (AI): Time of low wind chill

(The last 16 entries were added in version 1.9.1)

  • 35 (AJ): High dew point
  • 36 (AK): Time of high dew point
  • 37 (AL): Low dew point
  • 38 (AM): Time of low dew point

(Last 4 entries added in version 1.9.2 beta build)

(These last three entries were added from 1.9.2 Build 1004)

Finally, added in version 1.9.3 build 1036 (these only show valid values if appropriate sensors exist)

  • 42 (AQ): High solar radiation
  • 43 (AR): Time of high solar radiation
  • 44 (AS): High UV Index
  • 45 (AT): Time of high UV Index

(Cumulus MX uses all fields as listed above)

Example of the file

An extract of a few lines of the dayfile.txt

01/08/11,19.3,61,10:22,12.5,06:58,23.8,14:49,1014.26,20:46,1018.83,09:28,0.0,00:00,0.0,17.8,21.6,4.6,10:44,36,14:14,86,01:56,3.56,8.9,23.8,14:49,23.1,14:50,12.3,06:59,0.0,00:00,12.5,06:58,11.3,00:16,6.9,14:34,354,2.0,1.5

02/08/11,16.1,20,16:55,14.7,06:45,24.2,13:54,1013.79,19:13,1015.65,11:14,0.0,00:00,0.0,18.9,13.7,8.0,15:55,42,20:42,85,06:50,2.79,4.9,24.2,13:54,24.3,13:55,15.1,06:40,0.0,00:00,14.7,06:45,14.8,11:59,7.0,21:09,57,1.0,1.7

03/08/11,14.5,36,17:23,14.9,05:50,24.6,14:46,1012.70,18:44,1015.99,08:34,0.0,00:00,0.0,19.4,17.2,4.8,16:04,50,14:38,79,07:04,3.05,5.8,24.6,14:46,25.4,14:47,15.0,05:50,0.0,00:00,14.9,05:50,14.2,20:01,8.9,00:16,32,0.8,1.9

04/08/11,17.7,16,15:43,14.1,06:20,25.3,15:06,1013.08,18:42,1015.31,08:28,0.0,00:00,0.0,20.2,19.4,8.1,14:12,52,18:20,92,06:55,3.30,9.1,25.3,15:06,26.8,14:55,14.9,06:20,0.0,00:00,14.1,06:20,15.8,14:55,12.5,06:25,36,1.0,2.9

05/08/11,16.1,32,12:52,14.2,06:12,22.2,14:07,1013.89,00:01,1016.36,09:43,0.0,00:00,0.0,18.6,21.6,5.2,13:00,62,15:57,87,06:11,3.30,8.4,22.2,14:07,23.5,14:10,14.8,07:19,0.0,00:00,14.2,06:12,15.4,10:33,12.0,06:03,34,0.9,1.3

06/08/11,16.1,309,11:15,14.3,05:29,22.4,17:12,1014.46,20:02,1016.97,10:38,0.0,00:00,0.0,18.4,19.2,5.5,16:21,55,13:33,92,05:20,2.79,7.9,22.4,17:12,23.3,18:17,15.1,06:09,0.0,00:00,14.3,05:29,14.2,18:12,10.9,10:38,32,1.1,1.3

07/08/11,17.7,342,13:24,12.9,05:47,24.1,14:53,1013.92,19:49,1016.43,09:36,0.0,00:00,0.0,18.4,19.1,6.3,14:06,48,12:45,89,05:36,3.30,9.0,24.1,14:53,24.6,15:48,13.3,05:47,0.0,00:00,12.9,05:47,14.6,15:52,10.7,11:33,11,1.6,1.7