Cumulus Wiki

Dayfile.txt: Difference between revisions

Page Discussion

Dayfile.txt (view source)

Revision as of 14:18, 26 January 2021

8,400 bytes removed ,  26 January 2021
m
Minor rewording
m (→‎When Cumulus is left running: correction of typo)
m (Minor rewording)
= When Cumulus is left running =
 
* Cumulus is frequently reading observations from your weather station, but these don't affect the daily log "dayfile.txt" as it is only updated once a day.
* There are no updates to dayfile.txt at any other times, but (for Cumulus 1 only) the contents of the file are read and processed for many of the display and edit menu options that can be selected from the main Cumulus 1 screen.▼
* Cumulus tracks the highs and lows in weather observations by comparing read values against those it has stored in [[Today.ini]], updating that file as required.
▲*{{Version Therebadge are1}}The no updates to dayfile.txt at any other times, but (fororiginal Cumulus 1software only)reads the contents of the file are read and processed'''dayfile.txt''' for many of the display and edit menu options that can be selected from the main Cumulus 1 screen.
**What is stored in today.ini is processed into what get stored as next line in dayfile.txt as described in next section.
*Both Cumulus 1 and MX have ways to edit '''dayfile.txt''' while Cumulus is running. Please see [[Correcting Extremes]] page for more information. Cumulus 2 does not even allow you to view this log file.
*(It also updates [[Alltime.ini]], [[Monthlyalltime.ini]], [[Year.ini]], and [[Month.ini]] when appropriate.
 
*(Periodically, the weather readings and derived values are stored in [[Standard log files]] which are set up for each month).
* Cumulus will not mind you accessing the daily log outside its software, except when it needs write access for processing end of day.
* If you do need to correct some rogue data in the log file, first take a copy and work on that copy, because any edits you do could muck up the specific format that Cumulus 1 or MX needs, there is a section on dealing with rogue data below. Only when you are absolutely sure that your edited copy meets all the constraints listed later, should you replace the original.
 
== When Cumulus processes the end of the (meteorological) day ==
*Back ups of both today.ini, and dayfile.txt, log files ''in their state after the end of day update'' are copied to the 'cumulus\backup\daily' folder, a maximum of only 9 daily sub-folders are retained.
 
==='''Optional'''===
* Some people require a copy of the local file to use on their web server. Consequently, after it has been updated they file transfer it to (or if their web server is local, copy it to) their web server. One way of doing this is [[Upload_Dayfile| described here]].
*Some people take a copy of the local file, and use it locally for other purposes. See [[#How you can use the daily log|How you can use the daily log section]] and also the [[[[Category::User Contributions|Cumulusutils]]]] link.
=== Populating a database table ===
 
*{{Version badge 1}} The [[ImportCumulusFile|article here]] describes a method that can be used with Cumulus 1 to mimic the contents of dayfile.txt in a database table. However, be aware that the later versions of that script have bee edited for MX, so you will need to use an older version of the script that fits the version of Cumulus 1 you are using.
*[[File:Badge vMx.png]] Please see [[MX_Administrative_Interface#Standard_Daily_Summary_Table]] section for details of how '''CumulusMX.exe''' has a standard option to insert a new row into a database table holding columns relating to the dayfile.txt fields and '''ExportMySql.exe''' can update the database table with past rows.
 
 
 
 
UPDATE BELOW
 
 
 
*There is a section at [[Cumulus_MX#Optional_Sections|Cumulus MX]] to explain how MX includes the ability to generate SQL for creating the database table, for updating it with past data, and to add a new row at the end of the day for the standard database table version of this daily summary log.
 
 
 
 
 
 
 
==== Using that table ====
 
[[File:Badge vMx.png]]
MX allows you to specify a different schema in the SQL it generates, but it does not offer that validation feature I just mentioned. I continue to use my Cumulus 1 script (with some changes as for example the weather diary works differently, I am querying SQLite from \CumulusMX\data\diary.db) now I use MX. In the MX standard functionality, you are limited to using web tags for your inputs, and some of those are affected by the end of day process. I tried various content in the custom EOD query, but it did not give me what I need for the scripts that produce my web pages.
 
MX automatically stores all end of month figures as log files, a feature that Cumulus 1 and 2 lacked, but as yet it does not actually use this extra data, and provides no simple facility to put what is in these files into database tables. There is no end-of-month selection for updates in MX, so you can't easily get as much from dayfile.txt as I do.
* For Cumulus MX, there are backups of 10 files, the extra ones are the weather diary and Cumulus.ini, that are copied to start-up folders in \CumulusMX\backup\, again there are only 9 kept, unless you back these up somewhere else.
 
If your weather station includes it own data logger, then Cumulus will read "archived records" from that log, and when it detects the archive log has moved to a new meteorological day, it will run the "end of day" process that moves what is in today.ini into a new line in dayfile.tx.
== How you can use the daily summary log file ==
 
* If you want to run scripts that use the daily summary log file, it is best if you take a copy first, you can ask Cumulus 1 to take a copy after each update by using the '''Daily''' box in the bottom left of the ''Sites/Options'' frame within the ''Internet'' options screen from the '''Configuration''' menu; that will safely take a copy of 'dayfile.txt' after it is updated. This has the advantage it happens even if Cumulus has been stopped and restarted and rollover is happening during catch-up and so not at usual rollover time according to the computer clock. See Cumulus 1 '''Help''' for information on using this feature, I add a redirection ">daily_batch.log" in the parameter box alongside so that any output from running the command file I specify in the main box is sent to a log file overwritten in each run; this enables me to see the reason for any failure.
* Cumulus MX has option to list files to be transferred once a day as part of rollover, so you can use that to generate your extra copy. This has the advantage it happens even if Cumulus has been stopped and restarted and rollover is happening during catch-up.
* A third party tool "Cumulus Toolbox" can also be used to copy/transfer files at a particular time. Note this cannot tell whether Cumulus has done its rollover at the normal time, or during catch-up.
* There are other ways to specify that when a file changes it is copied somewhere.
*The system routines that Cumulus uses to access dayfile.txt require exclusive use of that file, so if you have any other process trying to access that file when Cumulus restarts, when Cumulus processes end of the (meteorological) day, or when a relevant option is selected from View or Edit menus, either your external process or the Cumulus process may fail.
*Normally if you use any third-party packages like for example "Cumulusutils", the separator used in first line is assumed to be true for all lines.
*Some third-party tools have to be told what separator you use for dates, before they can read your dayfile.txt.
== Cumulus 2 ==
 
Does not provide any viewing functionality.
 
== Cumulus MX ==
 
There is a dayfile editor within the admin interface to edit this log file.
 
Only from MX version 3.6.0 has this been able to read the log file if it has some lines that were created using Cumulus 1 (with less than 46 fields). In the same version of MX, the number of fields in this log file was increased by 4 when Feels Like temperature was added. From that version all lines viewed in this editor will have 50 fields. The content of any field that was not in the line when it was created will be an empty string as far as this editor is concerned and any line edited and saved, therefore gains all these empty fields and will be stored as 50 fields until version 3.6.10.
 
From Emergency Version 3.6.12 (formally released in 3.7.0), all lines have 54 fields. The extras are Canadian Humidity Index (Humidex).
 
[[File:Badge vMx.png]]For Cumulus MX, when you select a line, both '''Edit''' and '''Delete''' buttons are enabled. There is no way of inserting new lines into dayfile.txt from within MX, nor of changing the dates in the file.
 
 
 
=== Editing inside MX ===
 
Pick Edit, click that and an editing dialog pops up, that does not let you change the line number nor the date, but all other fields show their current contents and you can overtype as necessary. Scroll down to see 2 buttons (how they are labelled depends on which version you are using), the left button ignores any edits you have made (it is labelled 'Close' or "Cancel" and simply does same effect as clicking the "x" in the top right corner), and the right hand button saves your changes (even if it is labelled 'Edit' rather than "Save" in the version you are using).
 
 
== Using the daily summary log on your web-site ==
 
If you upload the log file to your web site then (with the help of JavaScript) you can read the log file to obtain information to show on a web page. You could have a web page that shows a today.htm like table for the last 7 days by combining reading Cumulus web tags with reading from the log file.
 
Search the Cumulus support forum to see (for example) how others extract information from dayfile.txt to display on their web page a set of fields similar to those shown for 'Yesterday.htm' web page for other dates in the past, such as one year ago.
 
If you use a script to read what is in the daily summary log file into a database table, or use the functionality in Cumulus MX to upload automatically to a database table, then see [[Daily_Summary]] article for information about ALL of the ways to show values from this database table.
 
==Viewing summary figures for a month or period==
 
'''These notes apply to Cumulus 1 only'''
 
To view a summary of dayfile.txt for a month, calendar year or selected period, use ''This month'' (choose any month, default is month from your computer system date), ''This year'' (choose any year, default is year from your computer system date), or ''This period'' (choose any start and end dates, default is yesterday calculated from your computer system date), within the '''View''' [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|menu]].
*Remember the daily summary log has its records based on rollover to rollover days.
*In all cases they exclude the today details that are not stored on dayfile.txt until the end of day rollover.
*If you use 9am or 10am rollover, and choose '''View''' ''This period'' between midnight and your 9am/10am rollover any day your latest meteorological day is the yesterday in terms of your computer system date that 'This period' tries to display as its default day, and the display will initially appear blank.
*If you use 9am or 10am rollover, and choose '''View''' ''This month'' before your 9am/10am rollover on the first day of a new calendar month your latest meteorological month is different to your computer system month 'This month' tries to display as its default month, and the display will initially appear blank.
 
Most of the displayed results are for observations in the daily summary log, but a few parameters are not in that log and are derived from the monthly logs (e.g. average wind speed) or the weather diary (e.g. count of days with snow lying).
*On the screen displayed after selecting ''This month'', you can change the month and year required using the options at bottom left, click ''Update Display'' and the revised summary will be calculated.
*On the screen displayed after selecting ''This year'', you can change the year required using the options at bottom left, click ''Update Display'' and the revised summary will be calculated.
*On the screen displayed after selecting ''This period'', you can change the start date and end date then click ''Update Display'' to get the equivalent calculations displayed for part of a month or any other period.
 
 
Note differences between observation reports on View screens and those available as web tags.
*Date and time stamps:
**The day number shown on screen is the meteorological day (changing at rollover and that may be at midnight or 9am/10am) as that date appears in dayfile.txt;
**A time-stamp (with time and date) given in a web tag quotes a calendar date (always changing at midnight).
*Reported statistics example:
**The screen shows total number of dry or wet days in the month;
**The web tags report longest dry or wet period in the month.
 
= List of fields in dayfile.txt =
Sfws
5,838

edits

Retrieved from "https://www.cumuluswiki.org/a/Special:MobileDiff/7642"