Cumulus Wiki

Monthly log files

Revision as of 13:25, 30 May 2013 by Sfws (talk | contribs) (→‎What can Monthly log files be used for?: Correcting v1.9.3 menu addition)

Files are automatically created with names in the form <Month><Year>log.txt (for example, the file for August 2009 is called Aug09log.txt); an entry is made to the current month's file every ten (default value; you can change this on the station settings screen) minutes, recording the current sensor values. The file is in comma-separated format with one line per entry. For more information on these files see in the Cumulus help file, in the section “Data log file format”.

Viewing monthly log files

In the View menu, select Data logs, click Load... and your data files will be listed. If there is a problem see FAQ: I can't find my data files. Select the required monthly log and display it. This is a text viewer, and works best when at full screen, but even then you are likely to need to scroll both horizontally and vertically to look at all the figures. If you click on an individual figure the cell will be highlighted, but you cannot edit the figures on a view screen.

Viewing summary figures for a month

In the View menu, select This month, and a summary for the current calendar month will be calculated from the monthly log. Note that many parameters are different to those available as web tags.

To view for a different month (or if you use 9am/10am rollover, if current local time on the 1st day of a month is before rollover time so the log for the current calendar month has not been started), select the month and year required using the options at bottom left, click Update Display and the revised summary will be calculated. (Alternatively you can select This period and select a start date and end date then click Update Display to get the equivalent calculation for part of a month or any other period).

What can Monthly log files be used for?

After you have edited (or created any missing) Monthly log files, you can:

  • update the Alltime.ini by choosing all time records from the Edit menu. This is a text editor, and works best when at full screen. Click the Help button for detailed instructions on how all available monthly logs are used.
  • (if the created/edited monthly log is within the current year) update the year.ini by choosing This year's records from the Edit menu. This is a text editor, and works best when at full screen. Click the Help button for detailed instructions on how all available monthly logs for the year are used.
  • (if the created/edited monthly log is for the current month) update the month.ini by choosing This month's records from the Edit menu. This is a text editor, and works best when at full screen. Click the Help button for detailed instructions.
  • (from version 1.9.3) update the monthlyalltime.ini by choosing Monthly records from the Edit menu. This is a text editor, and works best when at full screen. Click the Help button for detailed instructions on how all relevant monthly logs for all years are used.

Note in each of above, you see the currently stored extremes, and you can load the monthly log(s) and dayfile.txt to view extremes calculated from those figures (as available) and optionally Copy the logged values (and associated date/time information) into the relevant ini file.

  • see everything shown (but not stored in the .ini files used for tracking extremes) in View menu displays for This month, This period, and This year (e.g. Air frosts, Gale days, Total Wind run figures are generated by reading the appropriate monthly log(s) and doing the necessary calculation).
  • create the relevant monthly and/or annual NOAA style report by choosing NOAA Monthly Report or NOAA Annual Report from the View menu, then select the required period using the selectors, click the Update Display button to see various statistics calculated from processing the monthly log files, and finally press Save button to store the new or amended report. (Note: You can generate a NOAA report for a period without a monthly log, if data for that period is present on dayfile.txt. Generation of complete NOAA reports requires information in dayfile.txt, and the monthly logs for period in question).
  • use create missing button in the dayfile.txt selection in the Edit menu; This is a text editor, and works best when at full screen. Cumulus will then look through all monthly log files and create approximate records for any missing dates in the dayfile if those dates have observations stored in the relevant monthly log. (It creates a file in the Cumulus folder called dayfileeditlog.txt which contains the entries it created). Click the Help button for more information. In past and current versions (including 1.9.3), create missing will not affect any records where some of the fields for a particular date are missing in the dayfile.txt (see workaround below).

Correcting any data problems

See FAQ: How do I correct my all-time, annual, or monthly records.

Editing within Cumulus

Highs and lows and the daily summaries (dayfile.txt) can all be edited within Cumulus, and the editors allow you to fetch the data from the monthly log files and dayfile.txt so that you can select data to replace the current ones. So once you have corrected the data in the monthly log files and dayfile.txt, you can correct the highs and lows using the editors.

Manipulation outside Cumulus

Any log files for previous months can be edited (outside Cumulus) with Cumulus running, and after editing them, you can do further fetch edits within Cumulus as described above. The current month log file can also be edited outside Cumulus, but you must stop Cumulus first.

Tips -- take a copy of the original log file before you work on it outside Cumulus (perhaps give the not to be touched copy a filename of "<Month><Year>log.csv").

Edit the original file using an editor that treats all fields as text [either any text editor, a CSV editor, or a spreadsheet program that can be instructed not to recognise special field (like date) types].

Rules -- Each line must contain the fields in correct sequence (since new versions/builds can add to number of fields, Cumulus will accept lines of various lengths without the more recent fields at the end). Nulls ',,' are not allowed in the line, so if you do not know the value for a particular field within the line, then type in a zero or -999 (check the format and range of numbers allowed for particular fields).

Important:

  • The date format uses two digits for the year and should be treated as text. Do not edit this file using software (e.g. Excel) with default of recognising formats, as it is likely to change the dates to have four figure years, and then Cumulus will no longer be able to use the file.
  • Times in these files are in the form hh:mm using the 24 hour clock and local time (system time).
  • Note that the fields related to solar data will only contain valid data if your station has the appropriate sensor(s).
  • Apart from bearings (stored as integers), most value fields are in format x.y using your system decimal notation. See Monthly log files: Example lines from the file.

Importing pre-Cumulus data

See FAQ: I've just installed Cumulus and it didn't download all the old data from my weather station

Given that monthly log files are used as input for updating the all-time, all-monthly (from version 1.9.3), and this year record extremes, for creating missing dayfile.txt entries, and for creating NOAA style reports, you may have some information from your weather station for a period before you first starting using Cumulus that you want to include.

There are some postings in the Support Forum about importing past data, and it is too wide a subject to fully cover here. Essentially match the fields listed in Monthly log files: List of fields in the file below with the fields you have available in your source.

There are some issues to consider. For example, wind directions might be reported as cardinal points (north, south etc) and need converting to bearings in (integer) degrees (spreadsheet packages provide look up functionality to do such conversions), and dates and times might need some pre-processing (spreadsheet packages usually have ability to select part of a text string and to cocatenate a number of strings) to convert them to text in the formats mentioned above.

Cumulus has various wind speed, temperature, and rainfall related fields, not directly available from weather station outputs. See, for example EasyWeather differences in FAQ: What do the various wind speeds in Cumulus correspond to with Fine Offset stations and other answers in the FAQ to pursue this; or add a post to the Support Forum for assistance. See the workaround described below for a way of getting Cumulus to generate (from your newly created monthly logs) apparent temperature, heat index, rain rate, wind run and other daily summary parameters that may not be available from your weather station.

The date that Cumulus first started tracking all-time records (<#recordsbegandate> see Webtags#Records), does not need to be updated for Cumulus to recognise earlier monthly log files in its data subfolder and in the various display options available from the Cumulus main screen. If you use the Edit functions to update the stored extremes as described above, any monthly logs found will be used, not just those after the initial Cumulus start date recorded in cumulus.ini, so the date will no longer reflect the start of analysis for all-time records. If you do choose to edit the start date, the method is to stop Cumulus, and amend the StartDate= line in cumulus.ini within the main Cumulus folder before re-starting Cumulus, but (in case you make a mistake) back up everything first!

If you leave the start date unchanged note that <#recordsbegandate> is on the default Cumulus recordsT.htm template with the label "Records began on", and you might want to amend that label wording on that template to indicate it is the date that Cumulus running began, rather than the date the station began operating and producing observations.

Using Monthly logs to deal with shorter (or incomplete) dayfile.txt records for particular dates

WORKAROUND if dates are present in monthly log and dayfile.txt but not all fields for that date exist in dayfile.txt

Example: records created by earlier version of Cumulus [to help you, the versions (not builds) at which fields were added are indicated below]. Second example: records imported prior to Cumulus processing them, so some calculated parameters (apparent temperature, heat index, rain rate, wind run) may not have been available from your weather station to insert in the log files.

One method is (not near rollover time) to (1) take a copy of dayfile.txt original as backup, (2) in original file delete any days with partial information (e.g. from Cumulus versions that created fewer fields), (3) use create missing in the dayfile.txt selection in the Edit menu option of Cumulus (note whilst datafile.txt normally calculates all parameters like minimums and maximums from very frequent samples of your weather station, the resolution of create missing is limited to the interval between logging records in the monthly log), (4) rename the amended dayfile.txt as dayfile(generated).txt, (5) create a new dayfile.txt and (6) use a text editor to merge the required fields from the new dayfile(generated).txt with all other fields from (with reading access only) the backup copy of the original file. (This method preserves the original as a backup so you can experiment with different merges and do some cross-checking).

List of fields in the file

  • 00: Date in the form dd/mm/yy or dd-mm-yy
  • 01: Current time
  • 02: Current temperature
  • 03: Current humidity
  • 04: Current dewpoint
  • 05: Current wind speed
  • 06: Recent (10-min) high gust
  • 07: Average wind bearing
  • 08: Current rainfall rate
  • 09: Total rainfall today so far
  • 10: Current sea level pressure
  • 11: Total rainfall counter
  • 12: Inside temperature
  • 13: Inside humidity
  • 14: Current gust (i.e. 'Latest')
  • 15: wind chill
  • 16: Heat index
  • 17: UV Index
  • 18: Solar Radiation

Added in 1.9.1:

  • 19: Evapotranspiration
  • 20: Annual Evapotranspiration
  • 21: Apparent temperature
  • 22: Current theoretical max solar radiation
  • 23: Hours of sunshine so far today

Added in 1.9.2:

  • 24: Current Wind bearing

Added in 1.9.3:

  • 25: RG-11 rain today

Added in 1.9.4 beta:

  • 26: Total Rainfall since midnight

Example lines from the file

Note that your field delimiters may be different, and your date delimiter too.


An extract of a few lines of the file (v.1.9.0)

Here logging is quarter hourly, the field delimiter is a semicolon, the decimal separator is a comma, and the date separator a dash. 

30-09-10;19:00;16,4;94;15,4;5,2;13,3;17;3,6;21,0;995,3;47,7;25,6;62;6,1;16,4;16,4;0,0;0 
30-09-10;19:15;16,4;94;15,4;5,6;11,2;12;18,0;24,0;995,0;50,7;25,6;62;7,2;16,4;16,4;0,0;0
30-09-10;19:30;16,2;94;15,2;7,9;15,8;355;7,2;25,8;994,3;52,5;25,7;62;12,2;16,2;16,2;0,0;0
30-09-10;19:45;16,0;94;15,0;9,9;19,4;7;7,2;27,9;993,3;54,6;25,7;62;14,8;15,8;16,0;0,0;0
30-09-10;20:00;15,9;94;14,9;12,4;20,9;354;7,2;30,0;993,0;56,7;25,7;62;19,4;15,4;15,9;0,0;0
30-09-10;20:15;15,8;94;14,8;8,4;15,8;349;14,4;32,7;993,4;59,4;25,8;61;12,2;15,8;15,8;0,0;0
30-09-10;20:30;15,4;94;14,4;13,8;33,1;317;28,8;40,5;993,7;67,2;25,8;61;23,4;14,7;15,4;0,0;0
30-09-10;20:45;15,1;94;14,1;20,3;34,2;356;7,2;43,8;992,3;70,5;25,8;60;29,5;13,8;15,1;0,0;0
30-09-10;21:00;15,3;94;14,3;20,2;35,6;358;10,8;46,8;991,0;73,5;25,8;60;28,1;14,0;15,3;0,0;0
30-09-10;21:15;15,3;95;14,5;16,6;31,7;358;10,8;49,5;991,4;76,2;25,8;60;20,9;14,3;15,3;0,0;0
30-09-10;21:30;15,3;94;14,3;14,0;27,0;324;18,0;54,3;992,3;81,0;25,8;60;15,8;14,5;15,3;0,0;0
30-09-10;21:45;15,2;94;14,2;13,0;25,6;323;10,8;57,9;992,3;84,6;25,8;59;24,5;14,5;15,2;0,0;0
30-09-10;22:00;15,0;94;14,0;16,7;31,7;312;10,8;60,6;993,0;87,3;25,8;59;23,4;13,9;15,0;0,0;0
30-09-10;22:15;14,9;94;13,9;16,0;30,6;357;10,8;63,0;991,6;89,7;25,8;59;20,9;13,9;14,9;0,0;0
30-09-10;22:30;14,9;94;13,9;17,6;31,7;3;3,6;63,3;990,6;90,0;25,8;59;19,4;13,7;14,9;0,0;0

this is an example of a file from 1.9.1, with solar data:

Here logging is increased to every five minutes, and using UK settings, the field delimiter is a comma, the decimal separator is a full stop, and the date separator a slash. 

22/04/11,10:25,8.1,96,7.5,13,20,138,0.0,0.0,1013.24,215.2,20.3,53,17,4.8,8.1,0.0,197,0.08,171.88,3.4,663,0.0
22/04/11,10:30,8.1,96,7.5,13,20,142,0.0,0.0,1013.28,215.2,20.2,53,11,4.8,8.1,0.0,216,0.08,171.88,3.3,673,0.0
22/04/11,10:35,8.1,96,7.5,12,18,142,0.0,0.0,1013.31,215.2,20.2,53,11,5.1,8.1,0.0,227,0.08,171.88,3.8,682,0.0

and from version 1.9.2:

Here logging is at the default interval, and the delimiters have their default settings for UK systems. 

01/10/11,04:40,18.0,75,13.5,5.0,6.7,169,0.0,0.0,1021.4,885.3,24.4,61,5.6,18.0,18.0,0.0,0,0.00,0.00,17.5,0,0.0,158
01/10/11,04:50,17.9,75,13.4,4.5,9.5,169,0.0,0.0,1021.4,885.3,24.4,61,2.0,17.9,17.9,0.0,0,0.00,0.00,17.5,0,0.0,180
01/10/11,05:00,17.8,75,13.3,5.1,9.5,177,0.0,0.0,1021.2,885.3,24.4,61,7.5,17.8,17.8,0.0,0,0.00,0.00,17.2,0,0.0,180
Retrieved from "https://www.cumuluswiki.org/index.php?title=Monthly_log_files&oldid=2667"