Monthly log files: Difference between revisions
m (→List of fields in the file: date formats) |
m (→Potential Name Confusion: rewrite of this section) |
||
| (146 intermediate revisions by 4 users not shown) | |||
| Line 1: | Line 1: | ||
[[Category:Cumulus Files]]Please see [[Standard log files]], [[Extra Sensor Files]], [[Air_Link_Log.txt]], [[Month.ini]], or [[Monthlyalltime.ini]], as relevant. |
|||
Files are automatically created with names in the form <Month><Year>log.txt (for example, the file for August 2009 is called <tt>Aug09log.txt</tt>); an entry is made to the current month's file every ''ten'' minutes (default value; you can change this on the station settings screen), recording the current sensor values. The file is in comma-separated format with one line per entry. |
|||
For more information on these files see in the Cumulus help file, in the section “Data log file format”. |
|||
==Correcting any data problems== |
|||
See [[FAQ#I.E2.80.99ve_just_installed_Cumulus.2C_and_it_didn.E2.80.99t_download_all_the_old_data_from_my_weather_station]], [[FAQ#How_do_I_correct_my_all-time_.28or_monthly.29_records.3F]] and [[FAQ#I_can.E2.80.99t_find_my_data_files.21]]. |
|||
After you have edited any Monthly log files, you can: |
|||
* update the [[Alltime.ini]] by choosing ''all time records'' from the '''Edit''' menu. This is a text editor, and works best when at full screen. Click the ''Help'' button for detailed instructions. |
|||
* (if the monthly log relates to the current year) update the [[year.ini]] by choosing ''This year's records'' from the '''Edit''' menu. This is a text editor, and works best when at full screen. Click the ''Help'' button for detailed instructions. |
|||
* (from version 1.9.3) update the [[monthlyalltime.ini]] by choosing ''all time records'' from the '''Edit''' menu. This is a text editor, and works best when at full screen. Click the ''Help'' button for detailed instructions. |
|||
* create the relevant monthly and/or annual NOAA style report by choosing ''NOAA Monthly Report'' or ''NOAA Annual Report'' from the '''View''' menu, then select the required period using the selectors, click the ''Update Display'' button to see various statistics calculated from processing the monthly log files, and finally press ''Save'' button to store the new or amended report. |
|||
Please be aware that all of the above links lead to documentation about files that may be described as ''monthly log files'' when reading individual Cumulus forum and wiki references. |
|||
===Editing in Cumulus=== |
|||
Only the current month's file can be edited from within Cumulus, choose ''This month's records'' from the '''Edit''' menu. This is a text editor, and works best when at full screen. Click the ''Help'' button for detailed instructions. |
|||
==Potential Name Confusion== |
|||
In the Cumulus '''View''' menu, choose ''This month'' or choose ''This period'' and then select the required period using the selectors to see various statistics calculated from processing the monthly log files. |
|||
===Manipulation outside Cumulus=== |
|||
Any log files for previous months can be edited with Cumulus running, unless you are trying to view them! |
|||
Those who used early versions of Cumulus will remember that the term '''Log File''' was originally used for [[Log.xml]], but Steve Loft later accepted that all the files with a [[:Category:Ini Files|.ini]] file extension could be described as log files, so he introduced the term [[Weather Diary]]. A bit later, [[Standard log files| the detailed data file]], began to be also called '''log files''', and the confusion grew. When [[Extra Sensor Files]] were added, these also became '''log files'''. The Wiki page now called "Standard Log Files" was first called "Log files", then became "Monthly Log Files" because "Log files" became a category. Then "Extra Sensor Files" were added and "standard log files" replaced "Monthly Log Files" to make it clearer which file held the standard values, and which file held values for extra sensors. |
|||
Tip -- take a copy of the file before you work on it outside Cumulus (perhaps call it <Month><Year>log.csv). |
|||
Edit the file using an editor that treats all fields as text (a text editor or a spreadsheet program that can be instructed ''not'' to recognise special field (like date) types). See |
|||
Steve Loft wrote the Cumulus software mainly for his own use, although after a while he let others use it and added features that others requested. Cumulus was developed in his spare time outside a full-time employment, and consistency and terminology were not issues he worried about. |
|||
Each line must contain the fields in correct sequence (i.e. nulls ',,' are not allowed in the line, but since new builds add to number of fields, Cumulus can accept lines without the more recent fields at the end). |
|||
During the development of the original Cumulus software (Cumulus 1 and 2) by Steve Loft, the name "monthly log" was most frequently used as a collective term for the last two (.ini) files linked above because those files tracked the monthly extreme records. |
|||
'''Important''': The date format uses two digits for the year. Do not edit this file using Excel or similar, as it is likely to change the dates to have four figure years, and then Cumulus will no longer be able to use the file. |
|||
When Steve Loft was working on his [[Cumulus_3_(MX)_beta_documentation|MX Beta]], he decided to implement within MX functionality the database schema that Mark Crossley had designed on [[ImportCumulusFile#Monthly_table_.28old.29|this]] page, where Mark selected the name '''Monthly''' for the table. David A Jamieson created versions 1.0 and 1.1 of ImportCumulusFile for creating a single table called '''dayfile''', but Mark extended it to cover ''realtime'' and what he called ''monthly log files''. That means the standard log can in MX optionally feed data into a database table called '''Monthly''' by default in MX. |
|||
Times in these files are in the form hh:mm using the 24 hour clock and local time (system time). Note that the fields related to solar data will only contain valid data if your station has the appropriate sensor(s). |
|||
Apart from bearings (stored as integers), most value fields are in format x.y using your system decimal notation. See [[Monthly_log_files#Example_lines_from_the_file]]. |
|||
===Importing pre-Cumulus data=== |
|||
Given that monthly log files are used as input for updating the all-time, all-monthly (from version 1.9.3), and this year record extremes, and for creating NOAA style reports, you may have some information for a period before you first starting using Cumulus that you want to include. |
|||
This database table terminology can be misleading as ''that database table is not organised by month in any way'', a single table contains every logged line that has been inserted into it by Steve's '''ExportMySQL.exe''', Mark's '''ExportToMySQL.exe''', or by '''CumulusMX.exe''' itself. In fact MX allows you to give this table another name, which it will then use in all SQL. |
|||
There are some postings in the support Forum about importing past data, and it is too wide a subject to fully cover here. |
|||
Essentially match the fields listed in [[Monthly_log_files#List_of_fields_in_the_file]] below with the fields you have available in your source. |
|||
Mark Crossley has taken over development of MX, the terminology "monthly log" is now being used for the first linked file (standard logs) instead of for '''month.ini''' the file that is truly logging monthly extremes. The name has been adopted simply because a new log file is started for each month. |
|||
There are some issues to consider. For example, wind directions might be reported as cardinal points (north, south etc) and need converting to bearings in degrees (spreadsheet packages provide ''look up'' functionality to do such conversions), and dates and times might need some pre-processing (spreadsheet packages usually have ability to select part of a text string and to cocatenate a number of strings) to convert them to text in the formats mentioned above. |
|||
For novices, there can be a feeling of confusion: |
|||
Cumulus has various wind speed, temperature, and rainfall related fields, not directly available from for example ''EasyWeather'' outputs. See [[FAQ#What_do_the_various_wind_speeds_in_Cumulus_correspond_to.2C_with_Fine_Offset_stations.3F]] and other answers in the FAQ to pursue this or visit the support Forum for assistance. |
|||
* The new practice leaves "month.ini" without a descriptive name |
|||
* Each "dayfile" line represents a new day so the filename matches the content, but a line in "Oct16log.txt" does not represent a month, and the file name does not contain the phrase "month" |
|||
* The names "Month.ini" and "Monthlyalltime.ini" do contain the phrase "month", they do contain monthly data in each line, and do log extreme records that clearly identify with a particular month |
|||
A future Wiki editor may decide on a different strategy, but at the moment the Cumulus Wiki has returned to the original Steve Loft terminology for each of the log files, for consistency with older articles in this Wiki, and the oldest posts on the forum. |
|||
The date that Cumulus first started operating, does ''not'' need to be updated for Cumulus to recognise earlier monthly log files in its ''data'' subfolder. However, the web tag associated with that date is on the default Cumulus ''recordsT.htm'' template, and you might want to amend the associated label wording to indicate it is the date that full Cumulus parameter logging/calculation began, rather than the date the station began operating. |
|||
=Enhancement never implemented= |
|||
If you do need to edit the start date, you can edit the ''StartDate='' line in cumulus.ini with Cumulus stopped, but back up everything first! |
|||
Steve Loft was several times asked if he could change the design of his software and what it stored in [[Standard log files]], [[Extra Sensor Files]] and the other detailed logging files. |
|||
==List of fields in the file== |
|||
All these files currently periodically store spot readings. Some of the competition in weather software, periodically store the highest and lowest since the previous line was stored. |
|||
* 00: Date in the form dd/mm/yy or dd-mm-yy |
|||
* 01: Current time |
|||
* 02: Current temperature |
|||
* 03: Current humidity |
|||
* 04: Current dewpoint |
|||
* 05: Current wind speed |
|||
* 06: Recent (10-min) high gust |
|||
* 07: Average wind bearing |
|||
* 08: Current rainfall rate |
|||
* 09: Total rainfall today so far |
|||
* 10: Current sea level pressure |
|||
* 11: Total rainfall counter |
|||
* 12: Inside temperature |
|||
* 13: Inside humidity |
|||
* 14: Current gust (i.e. 'Latest') |
|||
* 15: [[wind chill]] |
|||
* 16: [[Heat index]] |
|||
* 17: UV Index |
|||
* 18: Solar Radiation |
|||
Added in 1.9.1: |
|||
* 19: Evapotranspiration |
|||
* 20: Annual Evapotranspiration |
|||
* 21: [[Apparent temperature]] |
|||
* 22: Current theoretical max solar radiation |
|||
* 23: Hours of sunshine so far today |
|||
Added in 1.9.2: |
|||
* 24: Current Wind bearing |
|||
Added in 1.9.3 beta: |
|||
* 25: RG-11 rain today |
|||
One advantage of the alternative approach is that you never lose any extremes. Another advantage is the alternative allows people to report extremes for parts of days. For example, some people wanted 9 am to 9 pm reported separately to night time extremes, in other nations, the dividing time was 3 pm not 9 pm. |
|||
==Example lines from the file== |
|||
Steve Loft would respond he had made many design decisions that were not ideal, but the software was based on his own needs. If people wanted drastic changes, they should do what he had done, and write software that did what suited them. |
|||
Note that your field delimiters may be different, and your date delimiter too. |
|||
Yet it appears he did plan to implement this suggestion in MX, see [[Cumulusmx.db#Releases 3.0.0 to 3.11.4 inclusive]]. |
|||
An extract of a few lines of the file (v.1.9.0) |
|||
Here the field delimiter is a semicolon ,the decimal separator is a comma, and the date separator a dash. |
|||
30-09-10;19:00;16,4;94;15,4;5,2;13,3;17;3,6;21,0;995,3;47,7;25,6;62;6,1;16,4;16,4;0,0;0 |
|||
30-09-10;19:15;16,4;94;15,4;5,6;11,2;12;18,0;24,0;995,0;50,7;25,6;62;7,2;16,4;16,4;0,0;0 |
|||
30-09-10;19:30;16,2;94;15,2;7,9;15,8;355;7,2;25,8;994,3;52,5;25,7;62;12,2;16,2;16,2;0,0;0 |
|||
30-09-10;19:45;16,0;94;15,0;9,9;19,4;7;7,2;27,9;993,3;54,6;25,7;62;14,8;15,8;16,0;0,0;0 |
|||
30-09-10;20:00;15,9;94;14,9;12,4;20,9;354;7,2;30,0;993,0;56,7;25,7;62;19,4;15,4;15,9;0,0;0 |
|||
30-09-10;20:15;15,8;94;14,8;8,4;15,8;349;14,4;32,7;993,4;59,4;25,8;61;12,2;15,8;15,8;0,0;0 |
|||
30-09-10;20:30;15,4;94;14,4;13,8;33,1;317;28,8;40,5;993,7;67,2;25,8;61;23,4;14,7;15,4;0,0;0 |
|||
30-09-10;20:45;15,1;94;14,1;20,3;34,2;356;7,2;43,8;992,3;70,5;25,8;60;29,5;13,8;15,1;0,0;0 |
|||
30-09-10;21:00;15,3;94;14,3;20,2;35,6;358;10,8;46,8;991,0;73,5;25,8;60;28,1;14,0;15,3;0,0;0 |
|||
30-09-10;21:15;15,3;95;14,5;16,6;31,7;358;10,8;49,5;991,4;76,2;25,8;60;20,9;14,3;15,3;0,0;0 |
|||
30-09-10;21:30;15,3;94;14,3;14,0;27,0;324;18,0;54,3;992,3;81,0;25,8;60;15,8;14,5;15,3;0,0;0 |
|||
30-09-10;21:45;15,2;94;14,2;13,0;25,6;323;10,8;57,9;992,3;84,6;25,8;59;24,5;14,5;15,2;0,0;0 |
|||
30-09-10;22:00;15,0;94;14,0;16,7;31,7;312;10,8;60,6;993,0;87,3;25,8;59;23,4;13,9;15,0;0,0;0 |
|||
30-09-10;22:15;14,9;94;13,9;16,0;30,6;357;10,8;63,0;991,6;89,7;25,8;59;20,9;13,9;14,9;0,0;0 |
|||
30-09-10;22:30;14,9;94;13,9;17,6;31,7;3;3,6;63,3;990,6;90,0;25,8;59;19,4;13,7;14,9;0,0;0 |
|||
this is an example of a file from 1.9.1, with solar data: |
|||
Here using UK settings the field delimiter is a comma, the decimal separator is a full stop, and the date separator a slash. |
|||
22/04/11,10:25,8.1,96,7.5,13,20,138,0.0,0.0,1013.24,215.2,20.3,53,17,4.8,8.1,0.0,197,0.08,171.88,3.4,663,0.0 |
|||
22/04/11,10:30,8.1,96,7.5,13,20,142,0.0,0.0,1013.28,215.2,20.2,53,11,4.8,8.1,0.0,216,0.08,171.88,3.3,673,0.0 |
|||
22/04/11,10:35,8.1,96,7.5,12,18,142,0.0,0.0,1013.31,215.2,20.2,53,11,5.1,8.1,0.0,227,0.08,171.88,3.8,682,0.0 |
|||
and from version 1.9.2: |
|||
01/10/11,04:40,18.0,75,13.5,5.0,6.7,169,0.0,0.0,1021.4,885.3,24.4,61,5.6,18.0,18.0,0.0,0,0.00,0.00,17.5,0,0.0,158 |
|||
01/10/11,04:50,17.9,75,13.4,4.5,9.5,169,0.0,0.0,1021.4,885.3,24.4,61,2.0,17.9,17.9,0.0,0,0.00,0.00,17.5,0,0.0,180 |
|||
01/10/11,05:00,17.8,75,13.3,5.1,9.5,177,0.0,0.0,1021.2,885.3,24.4,61,7.5,17.8,17.8,0.0,0,0.00,0.00,17.2,0,0.0,180 |
|||
[[Category:Log Files]] |
|||
Latest revision as of 23:17, 31 December 2021
Please see Standard log files, Extra Sensor Files, Air_Link_Log.txt, Month.ini, or Monthlyalltime.ini, as relevant.
Please be aware that all of the above links lead to documentation about files that may be described as monthly log files when reading individual Cumulus forum and wiki references.
Potential Name Confusion
Those who used early versions of Cumulus will remember that the term Log File was originally used for Log.xml, but Steve Loft later accepted that all the files with a .ini file extension could be described as log files, so he introduced the term Weather Diary. A bit later, the detailed data file, began to be also called log files, and the confusion grew. When Extra Sensor Files were added, these also became log files. The Wiki page now called "Standard Log Files" was first called "Log files", then became "Monthly Log Files" because "Log files" became a category. Then "Extra Sensor Files" were added and "standard log files" replaced "Monthly Log Files" to make it clearer which file held the standard values, and which file held values for extra sensors.
Steve Loft wrote the Cumulus software mainly for his own use, although after a while he let others use it and added features that others requested. Cumulus was developed in his spare time outside a full-time employment, and consistency and terminology were not issues he worried about.
During the development of the original Cumulus software (Cumulus 1 and 2) by Steve Loft, the name "monthly log" was most frequently used as a collective term for the last two (.ini) files linked above because those files tracked the monthly extreme records.
When Steve Loft was working on his MX Beta, he decided to implement within MX functionality the database schema that Mark Crossley had designed on this page, where Mark selected the name Monthly for the table. David A Jamieson created versions 1.0 and 1.1 of ImportCumulusFile for creating a single table called dayfile, but Mark extended it to cover realtime and what he called monthly log files. That means the standard log can in MX optionally feed data into a database table called Monthly by default in MX.
This database table terminology can be misleading as that database table is not organised by month in any way, a single table contains every logged line that has been inserted into it by Steve's ExportMySQL.exe, Mark's ExportToMySQL.exe, or by CumulusMX.exe itself. In fact MX allows you to give this table another name, which it will then use in all SQL.
Mark Crossley has taken over development of MX, the terminology "monthly log" is now being used for the first linked file (standard logs) instead of for month.ini the file that is truly logging monthly extremes. The name has been adopted simply because a new log file is started for each month.
For novices, there can be a feeling of confusion:
- The new practice leaves "month.ini" without a descriptive name
- Each "dayfile" line represents a new day so the filename matches the content, but a line in "Oct16log.txt" does not represent a month, and the file name does not contain the phrase "month"
- The names "Month.ini" and "Monthlyalltime.ini" do contain the phrase "month", they do contain monthly data in each line, and do log extreme records that clearly identify with a particular month
A future Wiki editor may decide on a different strategy, but at the moment the Cumulus Wiki has returned to the original Steve Loft terminology for each of the log files, for consistency with older articles in this Wiki, and the oldest posts on the forum.
Enhancement never implemented
Steve Loft was several times asked if he could change the design of his software and what it stored in Standard log files, Extra Sensor Files and the other detailed logging files.
All these files currently periodically store spot readings. Some of the competition in weather software, periodically store the highest and lowest since the previous line was stored.
One advantage of the alternative approach is that you never lose any extremes. Another advantage is the alternative allows people to report extremes for parts of days. For example, some people wanted 9 am to 9 pm reported separately to night time extremes, in other nations, the dividing time was 3 pm not 9 pm.
Steve Loft would respond he had made many design decisions that were not ideal, but the software was based on his own needs. If people wanted drastic changes, they should do what he had done, and write software that did what suited them.
Yet it appears he did plan to implement this suggestion in MX, see Cumulusmx.db#Releases 3.0.0 to 3.11.4 inclusive.
