Cumulus Wiki

CumulusUtils

Revision as of 09:46, 27 May 2024 by HansR (talk | contribs) (→‎Batch file code of cumulusutils.bat)

Introduction

This is the first page of the CumulusUtils Wiki. From here you should be able to find everything you need to know for this tool.

If needed you can always go to the CumulusUtils Category page (see bottom of this page).

Prerequisites

This paragraph describes what is required to use CumuluUtils (To be modified/extended by user experience).

  1. A working CumulusMX environment on Windows, Linux (RPi) or MacOS
    1. On Linux (and probably MacOS too) the user must verify whether the application lshw has been installed.
    2. NOTE: no experience by the author on MacOS exists so be prepared to communicate
  2. At least 32 days of data This requirement has been dropped.
  3. For the Miscellaneous Charts more data - a year starting on jan 1 - is required
  4. For the logfiles:
    1. A consistent date format, mixed date formats in logfiles are not accepted by CumulusUtils From CMX v4 onwards this is enforced by CMX
    2. Consistent data and separator use See above
    3. If a legal date format in a logfile is used which is not implemented, just request it See above
  5. For longer data series check your locale creates consistent logfile entries and it will be useful to run the CMX supporting application CreateMissing.exe beforehand.

Goals

The goal of CumulusUtils (abbr: Cutils or CUtils) is to facilitate website creation for users of CumulusMX without exposure to coding in e.g. PHP or javascript. Having a meteo-website should not be privileged to users with great IT-skills. Making charts should not be demanding for programming skills in SQL or making it otherwise impossible to create charts without diving deep into the technique involved in CumulusMX (which actually is also the case with SQL). Note that this does not mean you can't use IT skills: the user configurable menu offers an opening to expand as far as you wish.

CumulusUtils positions itself as a configurable application for which the understanding of how to run it and how to configure it is the most important thing to know. For configuration CumulusUtils uses cumulusutils.ini file which resides in the CumulusMX directory. If that file does not exist, it will be created. All other files related to CumulusUtils are in the utils directory.

The charts are created using HighCharts using their non-commercial license. If you want to use the charts or the Website Generator of CumulusUtils, please make yourself acquainted with this license.

Assumptions

The user is assumed to have basic computer skills and knowledge about the Operating System and directory structure. The user must also be aware of the directory infrastructure of CumulusMX and needs to be aware of the meaning of the terms webroot and FTProot and Working Directory. The user should be able to understand and edit the ini files (both from CumulusMX and CumulusUtils). The user should understand the basic functioning of CumulusMX which means the correct availability of CumulusMX is a precondition for using CumulusUtils.

Some output files can be used with Cumulus 1 but the charting possibilities rely on the availability of the JSON datafiles of CumulusMX. Therefore the use of CumulusUtils in combination with Cumulus 1 is limited. The website generator in combination with Cumulus 1 is not advised.

Output

The output of CumulusUtils consists of mainly of text files (extension: .txt) which are generated on demand. In Cutils idiom modules. These text files can be incorporated in a website the user has or is making. Ultimately when using the website generator feature, CumulusUtils generates a complete website, uploaded to the user domain and extendable through a user defined menu. When using the ChartsCompiler, the user can define his own charts and place the output where he wants just like other modules.

All output of CumulusUtils is written as UTF8 encoded text files.

Installation

  1. CumulusUtils is available in a distribution which can be downloaded from the forum. In the CumulusMX directory the user must create a directory utils (case dependent) and must copy the files of the distribution (including the subdirectories) to that directory.
  2. Then, on the domain for the website, the user needs to make two directories in the webroot: 'lib' and 'css'. In addition the distribution directory CUicons must be copied as a whole - with contents - to the webroot.
  3. Updating CumulusUtils is done by completely overwriting the contents of the installation directory utils as if it were an installation. If you want to save the old installation, make a copy of the utils directory.
  4. NOTE a seemingly important issue: the datafiles (the naming and the contents) are dependent for their format on the locale / country setting of your machine. If you install and run from scratch and do not bother CMX will take the country setting from the settings of the computer. You may also give the country setting on the commandline when starting CMX. However, it is important to know that CumulusUtils does not handle datafiles with mixed languages and produces lots of errors. When starting with a new install this is not a big deal. But if you have years of old data which you wish to analyse as well you must consider carefully which country setting of CMX is required. Note that the language setting of CumulusUtils is for display/language handling only and does not affect the reading of the data in any way apart for the monthly log file name. If you use a different locale than the one you use for CMX, the please fill in the parameter MonthsOfMiracleAndWonder (section [General]) with the abbreviated filenames of the locale you use for CMX.

When installed you are ready to run. The first run after an install or an update MUST be without Thrifty.

So after installing you have:

The directory structure for utils on the CumulusMX machine is: 

 utils			(contents in the distribution)
   |--- bin		(in the distribution (from 4.7.0 and up))
   |--- utilslog	(Created by CumulusUtils)

The directory structure for utils on the Website is: 

 WebRoot		(Content uploaded by CumulusUtils)
   |--- lib		(JavaScript libraries, uploaded by CumulusUtils during a run)
   |--- css		(css files, uploaded by CumulusUtils during a run)
   |--- CUicons        (required to be manually copied once)

Updating CumulusUtils

Updating CumulusUtils is done by completely overwriting the contents of the installation directory utils as if it were an installation. If you want to save the old installation, make a copy of the utils directory. The first run after an install or an update MUST be without Thrifty.

Informational files

In the distribution there are some files which are examples or defaults. When used, rename them (and possibly copy manually to the webroot). No automatic copying of these files is done by CumulusUtils.

Beside the libraries the distribution contains:

  1. CUserAbout-example.txt : as an example of the user about file - when used rename to CUserAbout.txt
  2. CutilsMenu-example-for-use.def : as an example of the menu structure - when used rename to CutilsMenu.def
  3. CutilsCharts-default-for-use.def : when used rename to CutilsCharts.def
  4. CutilsCharts-examples.def : not intended for direct use, contains some more elaborate examples of the ChartsCompiler possibilities

To avoid confusion it is left to the user to edit and maintain the files for use either on the website or on the CumulusMX machine.

Manually installing without FTP

NOTE: Not using the automatic upload system is disadvised. Because CumulusUtils is datadriven it requires the JSON files and the realtime.txt and realtimegauges.txt from CMX. Besides those, if you are using the UserAskedData feature of CumulusUtils there will be additional JSON files required for the server and without the upload module your will need a way to get them there. There is no copy feature like CMX has as CumulusUtils is designed as a website tool and an upload method is implicitely assumed present. However, manual handling can be done but is not advised.

All required installation and generated files are present in the utils directory and copied with FTP or PHP upload to the server during a run of CumulusUtils.

If you don't use FTP or PHP upload you will have to copy manually.

Files involved

The list of installation files required and their destination is below. The generated Javascript files are included in the table:

Distribution File Destination Remark
index.html <webroot>
CUgauges.js <webroot>/lib Despite the same name this file is very different from the one in CMX
CUlanguage.js <webroot>/lib
HighchartsDefaults.js <webroot>/lib
suncalc.js <webroot>/lib
CUtween.min.js <webroot>/lib
CUsteelseries.min.js <webroot>/lib
CURGraph.rose.js <webroot>/lib
CURGraph.common.core.js <webroot>/lib
CUgauges-ss.css <webroot>/css
Generated File Destination Remark
cumulusutils.js <webroot>/lib
HighchartsLanguage.js <webroot>/lib
*.txt <webroot>
airlinkdataIn2p5.json <webroot>
airlinkdataIn10.json <webroot>
airlinkdataOut2p5.json <webroot>
airlinkdataOut10.json <webroot>
extrasensorsdata.json <webroot>
customlogsRecentdata.json <webroot>
customlogsDailydata.json <webroot>
CUserdataRECENT.json <webroot>
CUserdataDAILY.json <webroot>
CUserdataALL.json <webroot>
Realtime Files Destination Remark
realtime.txt <webroot> Served by CMX [1]
realtimegauges.txt <webroot> Served by CMX [1]
airlinkrealtime.txt <webroot> When the AirLink device is configured [2]
extrasensorsrealtime.txt <webroot> When Extra Sensor devices are configured [2]
customlogsrealtime.txt <webroot> When Extra Sensor devices are configured [2]
meteocamrealtime.txt <webroot> When Extra Sensor devices are configured [2]

[1] : These files may be served by CMX in another directory (just like the CMX JSON data files). In that case the relative location is specified in the inifile parameter CumulusRealTimeLocation.
[2] : these files are generated in the utils directory and need to be configured as Extra Webfiles in CMX with the ticks: Realtime, Process and FTP. Note that the generated file has the webtags and remains local, the processed files contains the values and must be send to the <webroot>.

Running CumulusUtils

CumulusUtils is - up to v6.x.y - a mono executable. From v7.0.0 and up it runs under dotnet (.NET 8).

Both environments are very different and not interchangeable. However running CumulusUtils is pretty similar on both environment and mono and dotnet can co-exist on the same machine. See the CMX installation on how to install either mono or dotnet. CumulusUtils assumes the correct installation of either environment.

Mono is an open source implementation of Microsoft's .NET Framework based on the ECMA standards for C# and the Common Language Runtime. CumulusUtils runs on any operating system CumulusMX runs on and it MUST run in the CumulusMX directory (as working directory). Running CumulusUtils is done from the commandline in a command window (under any OS).

CumulusUtils takes one or more commandline parameters (in short: commands) and must be like (see NOTE 1 below; square brackets means optional so DON'T TYPE THEM): 

               utils/bin/cumulusutils.exe [command]  (the mono syntax - this assumes mono is already active)

               dotnet utils/bin/cumulusutils.dll [command]  (the .NET syntax - dotnet is obligatory on the commandline. Note the dll)

Below is described for mono, when using .NET, please change the command as above. Note you will have to carry this on in scripts as well. Using the dotnet command in crontab requires using the explicit path to the dotnet command or defining the PATH environment variable in the crontab script which does not know the environment (a bit awkward).

If no commands are given the application responds with: 

               CumulusUtils : No Arguments nothing to do. Exiting. See Manual.
               CumulusUtils Usage : utils/bin/cumulusutils.exe [args] (args case independent):
                 utils/bin/cumulusutils.exe
                     [ SysInfo ] [ Forecast ] [ StationMap ] [ UserReports ] [ MeteoCam ]
                     [ pwsFWI ][ Top10 ][ Graphs ][ Yadr ][ Records ]
                     [ NOAA ][ DayRecords ][ AirLink ][ UserAskedData ]
                     [ CompileOnly ] [ ExtraSensors ] [ CustomLogs ]
                     | [ Thrifty ] 

               OR (in case you use the website generator):
                  utils/bin/cumulusutils.exe [ Thrifty ] Website

All modules mentioned in the command will be described in their own pages here on the wiki and when available will be linked from here.

The commands represent the modules the user is asking for to be generated. The output is generated to the utils directory in the CumulusMX directory. When asking for the generation of the website, the user not only gets all modules available but also an index.html file accompanied with some javascript infrastructure. Everything together then is the website which, when copied to the webroot, is ready for use.

Commands to CumulusUtils - the module names - are case independent but the author prefers Upper Camel Case (or Pascal Case).

Copying to the webroot can also be done automatically through the FTP account of the user, much like CumulusMX itself copies e.g. the realtime.txt or the datafiles to the webroot. The FTP account and domain used are the same as present in the Cumulus.ini configuration. The PHP upload as exists in CMX can also be used with CUtils and it works exactly the same.

Considerations with CMX configuration

Please note that CumulusUtils requires live data from CumulusMX to display what you expect. To accomplish that you need to configure CMX as follows:

The things it is looking for are realtime.txt and realtimegauges.txt. Those are sent by CMX to the directory you configure in Settings=>Web/Upload Site. 

In Settings=>Internet Settings=>Interval Configuration=>Normal Interval Settings: Tick both options and set the interval time to enable interval Upload
In Settings=>Internet Settings=>Interval Configuration=>Standard File settings: Disable both options (for CumulusUtils, if you have websites which use those, leave it enabled)
In Settings=>Internet Settings=>Interval Configuration=>Graph File Settings: Enable all (only Upload, use Create Local only when needed) except the following:
  availabledata.json, dailyrain.json, dailytemp.json, sunhours.json, airquality.json, extra*.json, soil*.json, user*.json,co2*.json and leaf*.json

In Settings=>Internet Settings=>Interval Configuration=>Daily Graph File Settings: Enable all except alldailydegreedaydata.json and alltempsumdata.json

In Settings=>Internet Settings=>Interval Configuration=>Real time Interval Settings: Tick both and set interval time to enable realtime Upload
In Settings=>Internet Settings=>Interval Configuration=>Real time Interval Settings: Tick both realtime.txt and realtimegauges.txt for Upload, use Create Local only when needed.

In Settings=>Web/Upload Site=>General Settings: Tick UTF-8 encoding

And if you use the ChartsCompiler (or plan to use, you may be more selective later when understanding what and how):

After selecting the required tables you need to select the variables they may contain: 

In Settings=>Station Settings=>Graphs=>Data Series Visibility: Tick all

NOTE: You may disable some fields later when you are more acquainted with the system. Disabling tables and field can be especially useful when you are worried about size of transfer and provider limits.

If you start using the ChartsCompiler, you may need the UserAskedData command to CumulusUtils. See the ChartsCompiler.

You may want to read about and understand the CumulusRealTimeLocation

When and why to run

Running CumulusUtils results in output which represents a static view of the data in a table or graphic format for display on your website (it may even create a complete website). The fact that the output is a static view requires the output to be regenerated when new data is available. Rerunning CumulusUtils is also required when you change anything in the configuration and can't wait to see that change reflected on the site. In general: changes in data and configuration require a rerun.

Reruns for changes of data are typically once per day, just after rollover time. In that sense it is like running a new query.

There are some exceptions to the daily rule: Sysinfo and UserAskedData

  1. Sysinfo: a rerun with Sysinfo is required when you wish to update the system information which can be a requirement from every 10 minutes to once per day
  2. UserAskedData: provides data for charts which are otherwise not available. Again this is at a user desired frequency which is recommended to be the FTP frequency defined in Cumulus.ini (parameter UpdateInterval in section [FTP site])

RPi: Using crontab

Automating the run of CumulusUtils on the Raspberry Pi is typically done through crontab. Below is the crontab of the author of CumulusUtils to be used as an example!

For dotnet
 05 1 * * *  cd /home/CumulusMX; /usr/share/dotnet/dotnet utils/bin/cumulusutils.dll thrifty website
 1-51/10 * * * *  cd /home/CumulusMX; /usr/share/dotnet/dotnet utils/bin/cumulusutils.dll sysinfo UserAskedData UserReports
For mono
 15 1 * * *  cd /home/CumulusMX; utils/bin/cumulusutils.exe thrifty website
 1-51/10 * * * *  cd /home/CumulusMX; utils/bin/cumulusutils.exe sysinfo UserAskedData UserReport

Windows: Using the scheduler

How to create a scheduled task

Prior to doing this we need to also create the script that the task will use. Open notepad or your favourite text editor (preferred use: Notepad++).

You can copy paste the below and save as cumulusutils.bat (It can be a filename of choice but must have extension of .bat, if using windows notepad, underneath the File name: option, will need to set Save as type: option to All Files (*.*). This way when the File name: is set as cumulusutils.bat it will save it as that and not cumulusutils.bat.txt)

Batch file code of cumulusutils.bat

For v6 and below (running under mono):

REM This batch file script checks for the process CumulusMX.exe is running, if yes, then runs the "cumulusutils.exe", otherwise exit

REM Set the current working session to the folder C:\CumulusMX

  cd\

  cd C:\CumulusMX

REM Check for CumulusMX.exe is running, if yes goto :runutils, otherwise goto :end

  tasklist /FI "IMAGENAME eq CumulusMX.exe" 2>NUL | find /I /N "CumulusMX.exe">NUL

  if "%ERRORLEVEL%"=="0" GOTO runutils

  GOTO end

REM Set as working directory C:\CumulusMX and run cumulusutils.exe in folder C:\CumulusMX\utils\bin, with the desired command or commands as per the wiki

:runutils

  START /D C:\CumulusMX C:\CumulusMX\utils\bin\cumulusutils.exe Website

:end

  exit

NOTE: The command shown here is Website. The user can put any command he likes in here (with or without Thrifty) depending on what he wants to achieve. Possibly he has to create multiple tasks (e.g. UserAskedData specifically runs on a higher frequency than website). These commands then require their own batch files.

For v7 and up (running under dotnet):

REM This batch file script checks for the process CumulusMX.exe is running, if yes, then runs the "cumulusutils.exe", otherwise exit

REM Set the current working session to the folder C:\CumulusMX

  cd\

  cd C:\CumulusMX

REM Check for CumulusMX.exe is running, if yes goto :runutils, otherwise goto :end

  tasklist /FI "IMAGENAME eq CumulusMX.exe" 2>NUL | find /I /N "CumulusMX.exe">NUL

  if "%ERRORLEVEL%"=="0" GOTO runutils

  GOTO end

REM Set as working directory C:\CumulusMX and run cumulusutils.exe in folder C:\CumulusMX\utils\bin, with the desired command or commands as per the wiki

:runutils

  START /D C:\CumulusMX dotnet C:\CumulusMX\utils\bin\cumulusutils.dll Website

:end

  exit
Task creation process

Within the Task Scheduler Library

  1. Create “New Task…” (actions pane), or right click middle pane and choose “Create New Task” (not Create Basic Task).
    1. On the General tab of the new task (General information about the task).
    2. Set desired name for the Task (Example name CumulusUtils, can also set description if desired).
    3. Click “Change User or Group…”.
    4. Type in here the word system (click on “Check Names” if desired, to confirm it is correct and it should then display as SYSTEM in capitals).
    5. Click OK to use this user account.
    6. Note in Security options part of the new task creation process it should now state under. When running the task, use the following user account: NT AUTHORITY\SYSTEM
      (Convenient reason for this is that when the task runs, it runs is running as system account and so is not seen popping up on screen when it is running.)
    7. “Configure for:” not required to be changed, can be changed to version of operating system.
  2. On the Triggers tab of the new task (Setting when and how often the task runs).
    1. Click on “New”, to create a new launch time and frequency of the task.
    2. Begin the task: option should default to “On a schedule”, this is the desired option.
    3. Under the settings section choose “Daily” option.
    4. Start: date can be left as is, set the desired start time of the task. Set to “12:00:15 am”.
      (Reason for this setting with 15 second delay is trying not to compete with CumulusMX upload tasks).
    5. In the advanced part, enable Repeat task every: and choose desired interval.
      With the option for a duration of: default setting of “1 day” is used.
    6. Set Stop task if it runs longer than: to “30 minutes” (if the task runs longer than the time set, it is stopped. This applies to the individually triggered task, and not the overall routine of this scheduled task, with it’s repeating trigger times).
    7. Confirm that the Enabled options is ticked to make active this trigger routine.
  3. On the Actions tab (Setting what the task is to do).
    1. Click on “New…” to create a new action.
    2. Action: should default to “Start a program” this is correct.
    3. In the Program/Script: part, type the path to the program to run, or click “Browse” to navigate and select the program to run. (example C:\CumulusMX\batch_files\cumulusutils.bat).
  4. On the Conditions tab (setting other conditions).
    1. Under Power, untick Start the task only if the computer us on AC power
      (This then enables the task to start and run even if on battery. It will automatically untick Stop if the computers switches to batter power.)
  5. On the Settings tab (additional other settings for the task).
    1. Confirm ticked for Allow task to be run on demand.
    2. By default Stop the task if it runs longer than: is already ticket and set as 3 days.Change to 1 hour (the behaviour here is the same as instruction 3.f.)
    3. Confirm ticked for If the running ask does not end when requested, for it to stop.
    4. Default setting of “Do not start a new instance” Is OK.
  6. Click on OK down the bottom to complete and commit the creation of this new task.
Customisation

One can also customise/fine tune the task further by exporting it, default export format is XML. Then can edit in favourite text editor and reimport when done. May be useful in particular for fine tuning the repetition times of the task.

Additional remarks

  1. (on NOT Windows) When mono is already active (as daemon or from command line in another process) you can start any mono executable without having it prepended by the mono command. As normally (if you did not uninstall it) xsp4 is running as a daemon, mono is always available so you can run mono executables as normal linux executables. You won't notice the difference. However if you don't have the x bit set, you get access refused when trying to execute. Without the x bit set you have to prepend mono as in: mono utils/bin/cumulusutils.exe. Then it executes without the x bit set because then it is an argument to the mono command.
  2. CumulusUtils is written in C#, HTML and javascript. All HTML and javascript is embedded in the C# code and generated on execution of CumulusUtils. The cumuluscharts.txt charts library was based on the cumuluscharts.js originally distributed with CumulusMX, probably the version a bit before April 2020 and has been modified since. With the appearance of the ChartsCompiler the name cumuluscharts.txt has been preserved but the technique of creating the charts has been changed.
  3. As CumulusUtils itself runs in the Cumulus directory, the data directory (data) is one level below. CumulusUtils currently uses - a copy of - dayfile.txt, the monthly logfiles and incidentally the alltime.ini file. It also takes information from Cumulus.ini. All output is written to the utils/ subdirectory. All logs are created in the utils/utilslog/ subdirectory. It also uses the AirLink.log and the ExtraLog files in a similar way when the user has such devices.
  4. The output of CumulusUtils consists of UTF8 encoded text files. They can be brought to your website manually through some ftp tool (e.g. WinSCP) or by using the Extra Web Files functionality in the setting of Cumulus (deprecated) or by using the FTP facility.
  5. CumulusUtils can output for many languages. See the specific page on language in CumulusUtils.
  6. Development of CumulusUtils started in 9th of August 2019 with an initial commit of the Top10 written in C. Development changed rapidly to C# with an initial commit of Top10 on November 19 2019 of version 0.7.0. The rest of the history can be read in the release notes.
  7. Users can present themselves on the CumulusUtils usermap. Although optional for modules users, the author sees it as payment for the usage of the tool, so please put yourself on the map. For website users it is automatic and obligatory.
  8. In case of problems there exists Errorlogging.
  9. CumulusUtils is © Hans Rottier and can be used under the Creative Commons Attribution License. At some point the code will be in the public domain under a different license.

NOTE: comments to this series of articles is welcome either by using the Talk feature of the Wiki for that page, direct edit of the article (request an account) or by contacting the author of CumulusUtils HansR on the forum or through PM.

Retrieved from "https://www.cumuluswiki.org/index.php?title=CumulusUtils&oldid=11732"