Monthly log files: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search
(Expanding 'Correcting any data problems' based on experience with importing from EasyWeather)
Line 1: Line 1:
Files are automatically created with names in the form <Month><Year>log.txt (for example, the file for August 2009 is called <tt>Aug09log.txt</tt>); an entry is made to the current month's file every ten minutes (default value; you can change this on the station settings screen), recording the current sensor values. The file is in comma-separated format with one line per entry.
Files are automatically created with names in the form <Month><Year>log.txt (for example, the file for August 2009 is called <tt>Aug09log.txt</tt>); an entry is made to the current month's file every ''ten'' minutes (default value; you can change this on the station settings screen), recording the current sensor values. The file is in comma-separated format with one line per entry.
For more information on these files see in the Cumulus help file, in the section “Data log file format”.
==Correcting any data problems==
See [[FAQ#I.E2.80.99ve_just_installed_Cumulus.2C_and_it_didn.E2.80.99t_download_all_the_old_data_from_my_weather_station]], [[FAQ#How_do_I_correct_my_all-time_.28or_monthly.29_records.3F]] and [[FAQ#I_can.E2.80.99t_find_my_data_files.21]].
After you have edited any Monthly log files, you can:
* update the [[Alltime.ini]] by choosing ''all time records'' from the '''Edit''' menu.  This is a text editor, and works best when at full screen. Click the ''Help'' button for detailed instructions.
* (if the monthly log relates to the current year) update the [[year.ini]] by choosing ''This year's records'' from the '''Edit''' menu.  This is a text editor, and works best when at full screen. Click the ''Help'' button for detailed instructions.
* (from version 1.9.3) update the [[monthlyalltime.ini]] by choosing ''all time records'' from the '''Edit''' menu.  This is a text editor, and works best when at full screen. Click the ''Help'' button for detailed instructions.
* create the relevant monthly and/or annual NOAA style report by choosing ''NOAA Monthly Report'' or ''NOAA Annual Report'' from the '''View''' menu, then select the required period using the selectors, click the ''Update Display'' button to see various statistics calculated from processing the monthly log files, and finally press ''Save'' button to store the new or amended report.
 
===Editing in Cumulus===
Only the current month's file can be edited from within Cumulus, choose ''This month's records'' from the '''Edit''' menu.  This is a text editor, and works best when at full screen. Click the ''Help'' button for detailed instructions.
 
In the Cumulus '''View''' menu, choose ''This month'' or choose ''This period'' and then select the required period using the selectors to see various statistics calculated from processing the monthly log files.
===Manipulation outside Cumulus===
Any log files for previous months can be edited with Cumulus running, unless you are trying to view them!
 
Tip -- take a copy of the file before you work on it outside Cumulus (perhaps call it <Month><Year>log.csv).
 
Edit the file using an editor that treats all fields as text (a text editor or a spreadsheet program that can be instructed ''not'' to recognise special field (like date) types).  See
 
Each line must contain the fields in correct sequence (i.e. nulls ',,' are not allowed in the line, but since new builds add to number of fields, Cumulus can accept lines without the more recent fields at the end).


'''Important''': The date format uses two digits for the year. Do not edit this file using Excel or similar, as it is likely to change the dates to have four figure years, and then Cumulus will no longer be able to use the file.
'''Important''': The date format uses two digits for the year. Do not edit this file using Excel or similar, as it is likely to change the dates to have four figure years, and then Cumulus will no longer be able to use the file.


Times in these files are in the form hh:mm using the 24 hour clock and local time (system time). Note that the fields related to solar data will only contain valid data if your station has the appropriate sensor(s).
Apart from bearings (stored as integers), most value fields are in format x.y using your system decimal notation.  See [[Monthly_log_files#Example_lines_from_the_file]].
===Importing pre-Cumulus data===
Given that monthly log files are used as input for updating the all-time, monthly (from version 1.9.3), and this year record extremes, and for creating NOAA style reports, you may have some information for a period before you fist starting using Cumulus.
There are some postings in the support Forum about importing past data, and it is too wide a subject to cover here.
Essentially match the fields listed in [[Monthly_log_files#List_of_fields_in_the_file]] below with the fields you have available in your source.


==Format of the file==
There are some issues to consider. For example, wind directions might be reported as cardinal points (north, south etc) and need converting to bearings in degrees (spreadsheet packages provide ''look up'' functionality to do such conversions), and dates and times might need some pre-processing (spreadsheet packages usually have ability to select part of a text string and to cocatenate a number of strings) to convert them to text in the formats mentioned above.  Cumulus has various wind speed, temperature, and rainfall related fields, not directly available from for example ''EasyWeather'' outputs.  See [[FAQ#What_do_the_various_wind_speeds_in_Cumulus_correspond_to.2C_with_Fine_Offset_stations.3F]] and other answers in the FAQ to pursue this.
The date that Cumulus first started operating, does ''not'' need to be updated, for Cumulus to recognise earlier monthly log files in its ''data'' subfolder.  However, the web tag associated with that date is on the default Cumulus ''recordsT.htm'' template, and you might want to amend the associated label wording to indicate it is the date that full Cumulus parameter logging/calculation began, rather than the date the station began operating.  If you do want to edit the start date, you can edit the ''StartDate='' line in cumulus.ini with Cumulus stopped.
==List of fields in the file==


* 00: Date in the form dd/mm/yy
* 00: Date in the form dd/mm/yy
Line 21: Line 50:
* 13: Inside humidity  
* 13: Inside humidity  
* 14: Current gust (i.e. 'Latest')  
* 14: Current gust (i.e. 'Latest')  
* 15: Wind chill  
* 15: [[wind chill]]
* 16: Heat Index
* 16: [[Heat index]]
* 17: UV Index
* 17: UV Index
* 18: Solar Radiation
* 18: Solar Radiation
Added in 1.9.1:
* 19: Evapotranspiration
* 19: Evapotranspiration
* 20: Annual Evapotranspiration
* 20: Annual Evapotranspiration
* 21: Apparent temperature
* 21: [[Apparent temperature]]
* 22: Current theoretical max solar radiation
* 22: Current theoretical max solar radiation
* 23: Hours of sunshine so far today
* 23: Hours of sunshine so far today
Added in 1.9.2:
* 24: Current Wind bearing
* 24: Current Wind bearing
Added in 1.9.3 beta
Added in 1.9.3 beta:
* 25: RG-11 rain today
* 25: RG-11 rain today
   
   


Times in these files are in the form hh:mm using the 24 hour clock and local time (system time). Note that the fields related to solar data will only contain valid data if your station has the appropriate sensor(s).


==Example of the file==
==Example lines from the file==


Filename:  <nowiki>sept10log.txt</nowiki><br/>  Note that your field delimiters may be different, and your date delimiter too.  Here the field delimiter is a semicolon ,the decimal separator is a comma, and the date separator a dash.
Note that your field delimiters may be different, and your date delimiter too.   




An extract of a few lines of the file (v.1.9.0)
An extract of a few lines of the file (v.1.9.0)


Here the field delimiter is a semicolon ,the decimal separator is a comma, and the date separator a dash.
  30-09-10;19:00;16,4;94;15,4;5,2;13,3;17;3,6;21,0;995,3;47,7;25,6;62;6,1;16,4;16,4;0,0;0  
  30-09-10;19:00;16,4;94;15,4;5,2;13,3;17;3,6;21,0;995,3;47,7;25,6;62;6,1;16,4;16,4;0,0;0  
  30-09-10;19:15;16,4;94;15,4;5,6;11,2;12;18,0;24,0;995,0;50,7;25,6;62;7,2;16,4;16,4;0,0;0
  30-09-10;19:15;16,4;94;15,4;5,6;11,2;12;18,0;24,0;995,0;50,7;25,6;62;7,2;16,4;16,4;0,0;0
Line 60: Line 91:
  30-09-10;22:30;14,9;94;13,9;17,6;31,7;3;3,6;63,3;990,6;90,0;25,8;59;19,4;13,7;14,9;0,0;0
  30-09-10;22:30;14,9;94;13,9;17,6;31,7;3;3,6;63,3;990,6;90,0;25,8;59;19,4;13,7;14,9;0,0;0


this is an example of a file from 1.9.1, using UK settings, with solar data:
this is an example of a file from 1.9.1, with solar data:
 
Here using UK settings the field delimiter is a comma, the decimal separator is a full stop, and the date separator a slash.


  22/04/11,10:25,8.1,96,7.5,13,20,138,0.0,0.0,1013.24,215.2,20.3,53,17,4.8,8.1,0.0,197,0.08,171.88,3.4,663,0.0
  22/04/11,10:25,8.1,96,7.5,13,20,138,0.0,0.0,1013.24,215.2,20.3,53,17,4.8,8.1,0.0,197,0.08,171.88,3.4,663,0.0

Revision as of 12:51, 5 January 2013

Files are automatically created with names in the form <Month><Year>log.txt (for example, the file for August 2009 is called Aug09log.txt); an entry is made to the current month's file every ten minutes (default value; you can change this on the station settings screen), recording the current sensor values. The file is in comma-separated format with one line per entry. For more information on these files see in the Cumulus help file, in the section “Data log file format”.

Correcting any data problems

See FAQ#I.E2.80.99ve_just_installed_Cumulus.2C_and_it_didn.E2.80.99t_download_all_the_old_data_from_my_weather_station, FAQ#How_do_I_correct_my_all-time_.28or_monthly.29_records.3F and FAQ#I_can.E2.80.99t_find_my_data_files.21. After you have edited any Monthly log files, you can:

  • update the Alltime.ini by choosing all time records from the Edit menu. This is a text editor, and works best when at full screen. Click the Help button for detailed instructions.
  • (if the monthly log relates to the current year) update the year.ini by choosing This year's records from the Edit menu. This is a text editor, and works best when at full screen. Click the Help button for detailed instructions.
  • (from version 1.9.3) update the monthlyalltime.ini by choosing all time records from the Edit menu. This is a text editor, and works best when at full screen. Click the Help button for detailed instructions.
  • create the relevant monthly and/or annual NOAA style report by choosing NOAA Monthly Report or NOAA Annual Report from the View menu, then select the required period using the selectors, click the Update Display button to see various statistics calculated from processing the monthly log files, and finally press Save button to store the new or amended report.

Editing in Cumulus

Only the current month's file can be edited from within Cumulus, choose This month's records from the Edit menu. This is a text editor, and works best when at full screen. Click the Help button for detailed instructions.

In the Cumulus View menu, choose This month or choose This period and then select the required period using the selectors to see various statistics calculated from processing the monthly log files.

Manipulation outside Cumulus

Any log files for previous months can be edited with Cumulus running, unless you are trying to view them!

Tip -- take a copy of the file before you work on it outside Cumulus (perhaps call it <Month><Year>log.csv).

Edit the file using an editor that treats all fields as text (a text editor or a spreadsheet program that can be instructed not to recognise special field (like date) types). See

Each line must contain the fields in correct sequence (i.e. nulls ',,' are not allowed in the line, but since new builds add to number of fields, Cumulus can accept lines without the more recent fields at the end).

Important: The date format uses two digits for the year. Do not edit this file using Excel or similar, as it is likely to change the dates to have four figure years, and then Cumulus will no longer be able to use the file.

Times in these files are in the form hh:mm using the 24 hour clock and local time (system time). Note that the fields related to solar data will only contain valid data if your station has the appropriate sensor(s). Apart from bearings (stored as integers), most value fields are in format x.y using your system decimal notation. See Monthly_log_files#Example_lines_from_the_file.

Importing pre-Cumulus data

Given that monthly log files are used as input for updating the all-time, monthly (from version 1.9.3), and this year record extremes, and for creating NOAA style reports, you may have some information for a period before you fist starting using Cumulus. There are some postings in the support Forum about importing past data, and it is too wide a subject to cover here. Essentially match the fields listed in Monthly_log_files#List_of_fields_in_the_file below with the fields you have available in your source.

There are some issues to consider. For example, wind directions might be reported as cardinal points (north, south etc) and need converting to bearings in degrees (spreadsheet packages provide look up functionality to do such conversions), and dates and times might need some pre-processing (spreadsheet packages usually have ability to select part of a text string and to cocatenate a number of strings) to convert them to text in the formats mentioned above. Cumulus has various wind speed, temperature, and rainfall related fields, not directly available from for example EasyWeather outputs. See FAQ#What_do_the_various_wind_speeds_in_Cumulus_correspond_to.2C_with_Fine_Offset_stations.3F and other answers in the FAQ to pursue this. The date that Cumulus first started operating, does not need to be updated, for Cumulus to recognise earlier monthly log files in its data subfolder. However, the web tag associated with that date is on the default Cumulus recordsT.htm template, and you might want to amend the associated label wording to indicate it is the date that full Cumulus parameter logging/calculation began, rather than the date the station began operating. If you do want to edit the start date, you can edit the StartDate= line in cumulus.ini with Cumulus stopped.

List of fields in the file

  • 00: Date in the form dd/mm/yy
  • 01: Current time
  • 02: Current temperature
  • 03: Current humidity
  • 04: Current dewpoint
  • 05: Current wind speed
  • 06: Recent (10-min) high gust
  • 07: Average wind bearing
  • 08: Current rainfall rate
  • 09: Total rainfall today so far
  • 10: Current sea level pressure
  • 11: Total rainfall counter
  • 12: Inside temperature
  • 13: Inside humidity
  • 14: Current gust (i.e. 'Latest')
  • 15: wind chill
  • 16: Heat index
  • 17: UV Index
  • 18: Solar Radiation

Added in 1.9.1:

  • 19: Evapotranspiration
  • 20: Annual Evapotranspiration
  • 21: Apparent temperature
  • 22: Current theoretical max solar radiation
  • 23: Hours of sunshine so far today

Added in 1.9.2:

  • 24: Current Wind bearing

Added in 1.9.3 beta:

  • 25: RG-11 rain today


Example lines from the file

Note that your field delimiters may be different, and your date delimiter too.


An extract of a few lines of the file (v.1.9.0)

Here the field delimiter is a semicolon ,the decimal separator is a comma, and the date separator a dash.

30-09-10;19:00;16,4;94;15,4;5,2;13,3;17;3,6;21,0;995,3;47,7;25,6;62;6,1;16,4;16,4;0,0;0 
30-09-10;19:15;16,4;94;15,4;5,6;11,2;12;18,0;24,0;995,0;50,7;25,6;62;7,2;16,4;16,4;0,0;0
30-09-10;19:30;16,2;94;15,2;7,9;15,8;355;7,2;25,8;994,3;52,5;25,7;62;12,2;16,2;16,2;0,0;0
30-09-10;19:45;16,0;94;15,0;9,9;19,4;7;7,2;27,9;993,3;54,6;25,7;62;14,8;15,8;16,0;0,0;0
30-09-10;20:00;15,9;94;14,9;12,4;20,9;354;7,2;30,0;993,0;56,7;25,7;62;19,4;15,4;15,9;0,0;0
30-09-10;20:15;15,8;94;14,8;8,4;15,8;349;14,4;32,7;993,4;59,4;25,8;61;12,2;15,8;15,8;0,0;0
30-09-10;20:30;15,4;94;14,4;13,8;33,1;317;28,8;40,5;993,7;67,2;25,8;61;23,4;14,7;15,4;0,0;0
30-09-10;20:45;15,1;94;14,1;20,3;34,2;356;7,2;43,8;992,3;70,5;25,8;60;29,5;13,8;15,1;0,0;0
30-09-10;21:00;15,3;94;14,3;20,2;35,6;358;10,8;46,8;991,0;73,5;25,8;60;28,1;14,0;15,3;0,0;0
30-09-10;21:15;15,3;95;14,5;16,6;31,7;358;10,8;49,5;991,4;76,2;25,8;60;20,9;14,3;15,3;0,0;0
30-09-10;21:30;15,3;94;14,3;14,0;27,0;324;18,0;54,3;992,3;81,0;25,8;60;15,8;14,5;15,3;0,0;0
30-09-10;21:45;15,2;94;14,2;13,0;25,6;323;10,8;57,9;992,3;84,6;25,8;59;24,5;14,5;15,2;0,0;0
30-09-10;22:00;15,0;94;14,0;16,7;31,7;312;10,8;60,6;993,0;87,3;25,8;59;23,4;13,9;15,0;0,0;0
30-09-10;22:15;14,9;94;13,9;16,0;30,6;357;10,8;63,0;991,6;89,7;25,8;59;20,9;13,9;14,9;0,0;0
30-09-10;22:30;14,9;94;13,9;17,6;31,7;3;3,6;63,3;990,6;90,0;25,8;59;19,4;13,7;14,9;0,0;0

this is an example of a file from 1.9.1, with solar data:

Here using UK settings the field delimiter is a comma, the decimal separator is a full stop, and the date separator a slash.

22/04/11,10:25,8.1,96,7.5,13,20,138,0.0,0.0,1013.24,215.2,20.3,53,17,4.8,8.1,0.0,197,0.08,171.88,3.4,663,0.0
22/04/11,10:30,8.1,96,7.5,13,20,142,0.0,0.0,1013.28,215.2,20.2,53,11,4.8,8.1,0.0,216,0.08,171.88,3.3,673,0.0
22/04/11,10:35,8.1,96,7.5,12,18,142,0.0,0.0,1013.31,215.2,20.2,53,11,5.1,8.1,0.0,227,0.08,171.88,3.8,682,0.0

and from version 1.9.2:

01/10/11,04:40,18.0,75,13.5,5.0,6.7,169,0.0,0.0,1021.4,885.3,24.4,61,5.6,18.0,18.0,0.0,0,0.00,0.00,17.5,0,0.0,158
01/10/11,04:50,17.9,75,13.4,4.5,9.5,169,0.0,0.0,1021.4,885.3,24.4,61,2.0,17.9,17.9,0.0,0,0.00,0.00,17.5,0,0.0,180
01/10/11,05:00,17.8,75,13.3,5.1,9.5,177,0.0,0.0,1021.2,885.3,24.4,61,7.5,17.8,17.8,0.0,0,0.00,0.00,17.2,0,0.0,180