NOTE: everything in the CumulusUtils Wiki pages related to ExtraSensors are provisional as long as ExtraSensors are not released as a whole
This paragraph describes what is required to use CumuluUtils (To be modified/extended by user experience).
- A working CumulusMX environment on Windows, Linux (RPi) or MacOS
- On Linux (and probably MacOS too) Mono 6 and up is required
- NOTE: no experience on MacOS exists
- At least 32 days of data
- For the Miscellaneous Charts more data is required
- A consistent date format in the logfiles
- Mixed data formats in logfiles are not accepted by CumulusUtils
- If a date format in a logfile is used which is not implemented, just request it
CumulusUtils positions itself as a configurable application for which the understanding of how to run it and how to configure it is the most important thing to know. For configuration CumulusUtils uses cumulusutils.ini file which resides in the CumulusMX directory. If that file does not exist, it will be created. All other files related to CumulusUtils are in the utils directory.
The user is assumed to have basic computer skills and knowledge about the Operating System and directory structure. The user must also be aware of the directory infrastructure of CumulusMX and needs to be aware of the meaning of the terms webroot and FTProot and Working Directory. The user should be able to understand and edit the ini files (both from CumulusMX and CumulusUtils). The user should understand the basic functioning of CumulusMX which means the correct availability of CumulusMX is a precondition for using CumulusUtils.
Some output files can be used with Cumulus 1 but the charting possibilities rely on the availability of the JSON datafiles of CumulusMX. Therefore the use of CumulusUtils in combination with Cumulus 1 is limited. The website generator in combination with Cumulus 1 is not advised.
The output of CumulusUtils consists of mainly of text files (extension: .txt) which are generated on demand. In Cutils idiom modules. These text files can be incorporated in a website the user has or is making. Ultimately when using the website generator feature, CumulusUtils generates a complete website, uploaded to the user domain and extendable through a user defined menu. When using the Charts Compiler, the user can define his own charts and place the output where he wants just like other modules.
All output of CumulusUtils is written as UTF8 encoded text files.
- CumulusUtils is available in a distribution which can be downloaded from the forum. In the CumulusMX directory the user must create a directory utils (case dependent) and must copy the files of the distribution (including the subdirectories) to that directory.
- Then, on the domain for the website, the user needs to make two directories in the webroot: 'lib' and 'css'. In addition the distribution directory CUicons must be copied as a whole - with contents - to the webroot.
- Updating CumulusUtils is done by completely overwriting the contents of the installation directory utils as if it were an installation. If you want to save the old installation, make a copy of the utils directory.
When installed you are ready to run. The first run after an install or an update MUST be without Thrifty.
So after installing you have:
The directory structure for utils on the CumulusMX machine is:
utils (contents in the distribution) |--- bin (in the distribution (from 4.7.0 and up)) |--- utilslog (Created by CumulusUtils)
The directory structure for utils on the Website is:
WebRoot (Content uploaded by CumulusUtils) |--- lib (Uploaded by CumulusUtils when in the utils dir) |--- css (Uploaded by CumulusUtils when in the utils dir) |--- CUicons (Manually copied once during setup)
Updating CumulusUtils is done by completely overwriting the contents of the installation directory utils as if it were an installation. If you want to save the old installation, make a copy of the utils directory. The first run after an install or an update MUST be without Thrifty.
CumulusUtils runs on any operating system CumulusMX runs on and it MUST run in the CumulusMX directory (as working directory). Running CumulusUtils is done from the commandline in a command window (under any OS).
CumulusUtils takes one or more commandline parameters (in short: commands) and must be like (square brackets means: optional):
If no commands are given the application responds with:
CumulusUtils : No Arguments nothing to do. Exiting. See Manual. CumulusUtils Usage : utils/bin/cumulusutils.exe [args] (args case independent): utils/bin/cumulusutils.exe [ SysInfo ] [ Forecast ] [ StationMap ] [ UserReports ] [ MeteoCam ] [ pwsFWI ][ Top10 ][ Graphs ][ Yadr ][ Records ] [ NOAA ][ DayRecords ][ AirLink ][ UserAskedData ] [ CompileOnly ] [ ExtraSensors ] | [ Thrifty ] All | CheckOnly
All modules mentioned in the command will be described in their own pages here on the wiki and when available will be linked from here.
Commands to CumulusUtils - the module names - are case independent but the author prefers Upper Camel Case (or Pascal Case).
Copying to the webroot can also be done automatically through the FTP account of the user, much like CumulusMX itself copies e.g. the realtime.txt or the datafiles to the webroot. The FTP account and domain used are the same as present in the Cumulus.ini configuration.
Considerations with CMX configuration
Please note that CumulusUtils requires live data from CumulusMX to display what you expect. To accomplish that you need to configure CMX as follows:
The things it is looking for are realtime.txt and realtimegauges.txt. Those are sent by CMX to the directory you configure in Settings=>Web/FTP Site.
In Settings=>Web/FTP Settings=>Interval Settings: Tick both options to enable realtime FTP In Settings=>Web/FTP Settings=>Interval Settings=>Standard File settings: Disable both options (for CumulusUtils, if you have websites which use those, leave it enabled) In Settings=>Web/FTP Settings=>Interval Settings=>Graph File Settings: Enable all (both Generate and Transfer) except availabledata.json and airquality.json
In Settings=>Web/FTP Settings=>Realtime Settings: Tick all
In Settings=>Web/FTP Settings=>General Settings: Tick only UTF-8 encoding
And if you use the ChartsCompiler (or plan to use, you may be more selective later when understanding what and how):
In Settings=>Web/FTP Settings=>Interval Settings=>Daily Graph File Settings: Enable all options except alldailydegdaydata.json and alltempsum.json otherwise, disable.
After selecting the required tables you need to select the variables they may contain:
In Settings=>Station Settings=>Graphs=>Data Series Visibility: Tick all
NOTE: You may disable some fields later when you are more acquainted with the system. Disabling tables and field can be especially useful when you are worried about size of transfer and provider limits.
You may want to read about and understand the CumulusRealTimeLocation
When and why to run
Running CumulusUtils results in output which represents a static view of the data in a table or graphic format for display on your website (it may even create a complete website). The fact that the output is a static view requires the output to be regenerated when new data is available. Rerunning CumulusUtils is also required when you change anything in the configuration and can't wait to see that change reflected on the site. In general: changes in data and configuration require a rerun.
Reruns for changes of data are typically once per day, just after rollover time. In that sense it is like running a new query.
There are some exceptions to the daily rule: Sysinfo and UserAskedData
- Sysinfo: a rerun with Sysinfo is required when you wish to update the system information which can be a requirement from every 10 minutes to once per day
- UserAskedData: provides data for charts which are otherwise not available. Again this is at a user desired frequency which is recommended to be the FTP frequency defined in Cumulus.ini (parameter UpdateInterval in section [FTP site])
RPi: Using crontab
Automating the run of CumulusUtils on the Raspberry Pi is typically done through crontab. Below is the crontab of the author of CumulusUtils to be used as an example!
15 1 * * * cd /home/CumulusMX; utils/bin/cumulusutils.exe thrifty website 9-59/10 * * * * cd /home/CumulusMX; utils/bin/cumulusutils.exe sysinfo UserAskedData 09 * * * * cd /home/CumulusMX; utils/bin/cumulusutils.exe sysinfo UserAskedData 0 9 * * * /home/cumubackup.sh > cumubackup.log 2>&1
Windows: Using the scheduler
- As CumulusUtils itself runs in the Cumulus directory, the data directory (data) is one level below. CumulusUtils currently uses - a copy of - dayfile.txt, the monthly logfiles and incidentally the alltime.ini file. It also takes information from Cumulus.ini. All output is written to the utils/ subdirectory. All logs are created in the utils/utilslog/ subdirectory. It also uses the AirLink.log and the ExtraLog files in a similar way when the user has such devices.
- The output of CumulusUtils consists of UTF8 encoded text files. They can be brought to your website manually through some ftp tool (e.g. WinSCP) or by using the Extra Web Files functionality in the setting of Cumulus (deprecated) or by using the FTP facility.
- CumulusUtils can output for many languages. See the specific page on language in CumulusUtils.
- Development of CumulusUtils started in 9th of August 2019 with an initial commit of the Top10 written in C. Development changed rapidly to C# with an initial commit of Top10 on November 19 2019 of version 0.7.0. The rest of the history can be read in the release notes.
- Users can present themselves on the CumulusUtils usermap. Although optional for modules users, the author sees it as payment for the usage of the tool, so please put yourself on the map. For website users it is automatic and obligatory.
- In case of problems there exists Errorlogging.
- CumulusUtils is © Hans Rottier and can be used under the Creative Commons Attribution License. At some point the code will be in the public domain under a different license.
NOTE: comments to this series of articles is welcome either by using the Talk feature of the Wiki for that page, direct edit of the article (request an account) or by contacting the author of CumulusUtils HansR on the forum or through PM.