Cumulus Wiki

Cumulus MX FAQ

Revision as of 18:59, 23 May 2020 by Sfws (talk | contribs) (→‎Using Custom SQL)

Introduction

If you are seeking information on why you might want to use Cumulus, then please read the About Cumulus article first. That is a general introduction to what Cumulus does and what might be its advantages over alternative software.

It is anticipated that the main documentation for Cumulus MX will be placed in this article and you should read that to get basic information about MX.

This article is to answer those awkward questions that will arise when you try to go beyond the "normal" installation or you need some trouble shooting style help. The questions have been placed into groups, but this is my first attempt at designing this article, so please feel free to make improvements that suit the way you search.


Request for assistance

If you, either need help on topics not covered in this article, or you see any messages that are not explained below, please update this article (you may need to ask for a login) or else post on the support forum in Cumulus Wiki Suggestions for updates and additions sub-forum. Thank you.

Background

Mark Crossley created this page on 26 October 2018 to help people with questions about MX where it works differently to Cumulus 1 and the Cumulus 1 FAQ did not help.

The original content covered device names (COMPORT in MX has different format to Port in C1), Windows 10 issues, and MONO issues. But my impression is nobody found this, and therefore nobody benefited from that text.

I started pasting in useful material, mostly from Steve Loft's posts in the forum about the beta version of MX, but as at that time I was not using MX I hoped somebody else would make sense of it and create a better layout. Of course nobody did.

When I did start using MX, I decided to edit other articles first, see Cumulus MX, Cumulus MX formal release versions, Webtags, Cumulus.ini, Dayfile.txt,, Php webtags, and various others.

So that left this page, and now I am starting on updating this article. If you read this, and believe you can help with completing this update, please do.

Troubleshooting - messages in MXdiags or in console/terminal output

Questions

Not watching the terminal or console window

Although the MX engine writes messages to the terminal or console (the terminology varies depending on the device) many people don't have a monitor connected to the device running MX (and if they do, they don't watch this output). Therefore it is easy to not to see messages, and not to know when MX is running or crashes.

One problem people have is stopping MX while it is busy, perhaps still doing catch-up from your station's logger after restarting, perhaps doing an end of day or end of month rollover, maybe in the middle of some other task. It is not so easy to see when MX is running, nor when it is busy. MX does so many tasks now, that picking a moment when it is not doing something is harder.

Although MX is designed assuming it is left running all the time, it is permissible to stop, and restart it, just do that as few times as possible (once a day if you must). MX will run on a variety of devices, so can run on devices that are cheap to leave on all the time.


I have a message about web tags

There is an article about Web tags, I have tried to keep that up to date. That may help, the most common errors seem to be:

  1. Using a web tag not available at the version you are using (the message in the mXDaiags should indicate which one).
  2. Using the wrong output formatters, as the article referenced above explains, the codes to use for MX are far more complicated.
  3. Using an input modifier that MX cannot match to data available (the message will talk about arrays).

I have a message about pressing Enter because a problem has occurred

This could cover a lot of problems. Make sure you don't have 2 copies of MX running and are not running anything else that accesses the same weather station or the same files as MX, because this message may relate to such a conflict.

Other problems that crash MX when you have just updated either MX or Mono, may relate to code added in update. For example 3.6.2 crashed in some contexts because of a new "FluentFTP.dll file", 3.6.3 reverted to the one used for 3.6.0. Other people have seen problems because of changes in how MONO worked on an update, although that was more of a problem with MX beta than later MX.

Problems with MX not accepting data when move from C1

Questions

Format of data files

Please see Log Files index page for more information. In general, you should have all the files (except log.xml from your C1 installation in your MX installation, and MX will be able to read them, although it may update content and thus stop C1 from recognising them. Ensure you have same culture as before (so don't change how dates/times and decimals are stored).

Ignoring earlier data

There are two contexts where new Cumulus MX users have reported problems stopping them seeing some earlier data they saw when they were using Cumulus 1 (C1).

  1. MX needs to read from Cumulus.ini the earliest date to process as it ignores any content for earlier dates. (Cumulus 1 just read all the files it could find in the data folder.
  2. When you first start running MX, it does not seem to be able to populate charts and recent history web tags for earlier times.

Dealing with rogue or missing input or derivatives

INPUT ISSUES: Sometimes a weather station will supply incorrect data bits that mean wrong numbers are input to Cumulus MX or that MX looking at byte level does not recognise what it is being sent.

DERIVATIVES ISSUES: Cumulus MX ignores any maximum and minimum values supplied by a weather station; it calculates these derived values for itself. For rainfall, MX does not use any rainfall readings supplied by the weather station, instead it works with a long term counter, and gets daily readings to output from subtracting one counter value (at end of required period) from another at start of day. It calculates further derived values like apparent temperature and feels like (and many others) by looking at various simultaneous readings (temperature, wind speed, and humidity for those two examples).

EXTREMES ISSUES: When MX is calculating those maximum and minimum derivatives, these are within an individual day, but they are compared against others in the same month (to update both the this month web tags and the monthly extremes web tags), against others in the same year (to update the this year web tags) and against all previous derivatives (to update the all-time web tags). I say update web tags, because that is how you view the extremes, but actually there is a log file for maximum and minimum in a day (today.ini), for this month (month.ini), for monthly-all-time (monthlyalltime.ini), for this year (year.ini), and for all time (alltime.ini). These log files are where MX stores the extremes and where you need to do corrections. For total rain this month, this year, maximum rain for this month in all years and highest rainfall (hourly, daily, monthly), the same log files are used. The highest daily wind run per month, per year, in the same month for all years and all-time is similarly stored in these log files and made available as web tags.

  1. Dealing with input problems depends on your weather station type
    • If there is a problem with input from a Davis weather station, stop MX, then it may be if you reset the Davis console, and the logger you use with your David, it might stop the bad data from then on, this won't correct the past data, but hopefully will stop the problem
    • If there is a problem with an old style Fine Offset, often with rain or pressure reported, try stopping MX, then resetting the Fine Offset (some have a small switch below a small hole on the back of the console so use a strong long pointed instrument with a small diameter, maybe a small nail or a sewing tool to initiate a reset). Disconnecting the USB connection and removing a console battery may also reset a Fine Offset, or there may be some combination of touches or switch pushes that does a reset.
    • For other weather station types, look in the forum for advice or read the station instructions.
  2. If the output values are frequently incorrect, MX provides some calibration settings in the admin interface. These vary dependent on the type of weather reading you want to correct. In general they use a formula like Output = Multiplier_2 X Input2 + Multiplier_1 X input + Offset so in the settings you can specify the multipliers and offset to use in that equation.
  3. If only a small numbers are incorrect, MX provides editors for the logs that store extremes (although for today only the rainfall can be edited). See the pages linked from Category:Log Files to discover more about each log and how to deal with rogue values. View and edit these onEdit tab of admin interface.
  4. You can correct any individual rogue value in a monthly log as MX (see viewer editor on Data logs tab of admin interface.
  5. As rainfall has a lot of different derivatives stored for it (hourly totals, daily totals, rain rate) it is quite complicated to correct, so there are a series of instruction in the C1 FAQ.

Problems with Cumulus MX use of MySQL database

CumulusMX includes in the admin interface the ability to create tables, and to update the tables. It works with both a predetermined schema (tables named realtime, monthly, and dayfile by default), and with a schema chosen by the Cumulus user. The package also includes the executable ExportMySQL.exe that can add past data to two of the tables (monthly and dayfile). These features allow people with no knowledge of the Structured Query Language (SQL) to start using databases, and problems are highly likely, see if the advice below helps.

Questions =

=What is my database name?

Cumulus MX does not create a database. You must have an existing database for MX to create tables. If you don't know how to create a database, don't use this facility. If you have created a database, or your web provider has provide you with a database, then you should know the database names, what user names have what access, and what password is needed.

Can I host a database locally?

Yes, the uploading that CumulusMX.exe and EXportMySQL.exe does will work on a local web server with a database as well as on a remote web server with a database. There are actually lots of packages that are free and will load Apache to run a web server, MySQL (or another database server) to allow you to use database tables, and a script language like PHP to allow you to generate web pages showing data from database queries. Then there are several packages that provides an easy way to manipulate a database (if you are using MySQL and PHP, then the most popular [but by no means the only] package has a name formed from those two with Admin added i.e.PHPMyAdmin; similar packages are available for other databases with the second part of name changing).

I'm updating my MX version and it says something about new columns

Although SQL will work with XML organised language (where each data item is labelled, but also has a relationship with other items), it is most often used with "relational databases". A relational database is defined as one that stores all the information in one or more tables consisting of rows and columns. Furthermore, the order of the columns must be defined by a schema that also states what format is used in each table cell, but the rows can be stored in any order, and the user cannot control that order.

When a table is being updated the SQL normally names the columns that are to be updated. For that update to work the columns must be there, and the number of columns must match the number of values being supplied. If a particular version of MX adds new columns for the update, then the new columns must be defined and added to the table before the update can run. The release announcement may provide a SQL script to run to add the new columns, otherwise you need to use whatever administration package is made available by your web server provider. If you have PHP and MySQL, then PHPMyAdmin is one package often provided, but there are several alternatives, that may offer more.

I am using Cumulus 1 and I have existing tables, but I want to try MX

  1. If the columns in your existing tables match those used by standard MX tables, then why not copy your existing tables (using a tool like PHPMyAdmin or whatever you have available) perhaps adding a "MX" suffix for the new name. Then in MX simply declare these table names via the Admin interface MySQL settings page. Set the host to the relevant name for your remote web server (or a local server), add username, password, alter settings to allow updates to leave your device, and alter settings on your web server to allow the MX device to update the database. Enable the relevant updates and MX should add more rows to the tables with your existing data.
  2. If there are extra columns in the MX schema compared to the schema you were using in Cumulus 1, follow instruction above as far as the necessary settings, but before you enable the updates either use any script provided in release post for when the extra columns were added, or use whatever administrative tool (it might be phpMyAdmin) is provide to add the extra columns. The enable the updates.
  3. If you want to use different columns to those MX uses (you don't have solar, or you want snow or extra sensor data to be in your database) then you need to use the Custom upload facilities that MX provides. To keep this answer short, I won't explain how here.

Using Custom SQL

Obviously to use Custom SQL you need some knowledge of SQL and this answer cannot teach you SQL. But the general format of an update instruction for a whole row is given below, ensure the command you supply to MX has all the parts shown. The instructions in {} are optional, they just make the SQL read better, parts in lower case need to be replaced by whatever is appropriate for your table, values are web tags enclosed in single quotes e.g. '<#windrun>', if there are any columns for which you don't have a value, use NULL without any quotes as this is SQL. 

 INSERT [INTO] database_table_name (primary_key_column_name, column_name_1, column_name_2, .... column_name_last) VALUES ('primary_key_value', 'web_tag_1', 'web_tag_2', ... 'web_tag_last');

Problems with web pages or NOAA reports

MX comes with a standard set of web pages. Although some look the same, these are different from the pages coming with C1. You may find that information labelled as for Cumulus 1 does apply to the MX web pages as for example they share the same cascading style sheet.

Questions

How to set up a web site

Please see Web site setup article.

My FTP is not working

  1. Check your firewall package, does it allow delete, insert, and rename operations
  2. Check your FTP server, does it allow delete, insert, and rename operations
  3. Test using your own FTP package, have you got host, username, password, and directory specified correctly
  4. The settings required are on the Internet settings screen within the admin interface.
  5. In the web/FTP site section, check the settings, and experiment with a different FTP protocol. For example MX can use standard FTP, or FTPS, and you can select whether to use TLS 1.2. Hopefuly whoever provides your web server has told you what FTP profile to use.
  6. In the web/FTP settings section, check all the settings, some determine what is automatically uploaded, other settings should initially obey the tips shown on the settings screen. One item to try is changing the setting for Active FTP mode.
  7. For most modern web pages, including those provided, use UTF-8 encoding description. If your web pages appear to be empty when they are uploaded, look near the front of such files where the encoding declarations will be found. Cumulus 1.9.4 build 1093 changed all the template pages from XHTML 1.0 Transitional using charset=iso-8859-1 encoding to HTML 5 using charset="UTF-8" encoding. It uses this without Byte Order Mark (BOM).
  8. Look up any FTP error codes on the web. Try Here

Directory related problems for FTP

The following pointers should help:

  • All web site file references are case sensitive. So if your FTP server directory is public_html, then 'Public_HTML' will not work!
  • On the Internet settings screen in Sites/Options the optional Directory parameter required is the path from your FTP root, to where the web pages are to be stored. It is not the Universal Resource Locator (URL) that you would specify to view the web page in a browser.
  • Relative paths (without a leading slash) are always safer than absolute paths (with a leading slash) and you may find that just specifying (for example) "public_html" works OK, but this all depends on how your ftp server and your ftp account is set up.
  • The host name should be associated with a particular entry point into the web server, so any directory entered into the settings should be a path relative to that entry point. The entry point for FTP may be different to the entry point for displaying web pages.

The bird image on standard web pages

  1. If you don't see this with standard web page set up then you have missed out the step of uploading all files and folders within the webfiles folder in the distribution just once to your web site. The contents of this folder are not used locally, you don't need to keep the folder after it has been uploaded to your web server (even if that is a local web server).
  2. If you don't want to see the bird image, please see C1 FAQ.

I want to remove the solar data from the supplied web templates

Please see C1 FAQ

How do I remove Inside Temperature graph or add solar/UV/sunshine?

Beteljuice has written a routine to change Highstock graphs in MX that is available here.

I want a different heading on standard web pages

These are built up from a combination of text in the template file and what you have entered on the station settings in text boxes just above the Options section.

The styling for these headings, including use of lower case, is defined in the Cascading Styling Sheet. You need some understanding of CSS to edit these.

I have a customised web template with yesterday web tags

Set this up via the Extra Web Files settings in the Admin interface. Use only static and yesterday web tags in your template, type the local path/file name in local column, tick the process column, tick the UTF-8 column if you declare that encoding in your template (see Customised templates for examples), tick the EOD column and put the path to where you want the web page to be created in remote column.

The sunrise and sunset times are wrong

  • You must enter DEGREES, MINUTES, and SECONDS, in the text boxes within the location section and select hemisphere, in the Station settings in the admin interface. See C1 FAQ for more advice.
  • Another possible cause might be your device system date and time is wrong.

The moon rise or moon set shows dashes

This is common, the lunar day does not match the earth day, so on some days the moon will either only rise or only set, while other days will see a moon rise and a moon set.

I want a moon image, but I don't see one

  1. Are you using an old version that does not include a moon image?
  2. Do you have the moon base image in your local files?
  3. Have you asked MX to generate by system drawing a masked image to show the current phase?
  4. Have you asked MX to upload this image?
  5. Is your FTP working, because that is only way the moon image can be uploaded to web server.

I modified the standard web pages, and since I updated to a new MX version I have lost my edits

In MX you can choose to either overwrite all files when there is a new release or to overwrite only files that you have not changed. In general, it is best not to modify the files provided with MX, instead copy them to a new folder and edit them there. For web templates, if they are in a different folder, you will need to specify them in the local column of Extra web files and turn off the option to upload standard web pages. For more information on customising web pages seeCustomised templates article.

I have problems with NOAA reports

  1. Editing Cumulus.ini without stopping MX can cause problems
  2. Changing the encoding on NOAA reports can cause problems, especially if you try to display them from a web page that does not share the same encoding.
  3. Making an error in the report naming parameters can cause problems, see NOAA_style_Report_Naming section.
  4. MX creates reports in a folder called "Reports" with an initial capital letter. That folder is part of the distribution in every MX release, and you must have that folder, not is equivalent in all lower case.
  5. There is a setting to determine whether NOAA reports are generated (at end of day), and another setting to control uploading to your web server (you need to specify the directory to upload to). Check these settings are all correct.

I have problems with my own web templates that Cumulus needs to process

  1. If your problems relate to web tags see earlier answer about error messages in MXDiags
  2. If you have problems with customising web pages, see reference in earlier answer
  3. If you have problems with selections in "Extra web files", see Extra_Web_Files.
  4. If you have problems with layout please see Web page layout article.

I am trying to use PHP Hypertext Pre-processor

PHP is a language for script that can do various tasks, but frequently it is used to prepare Hyper Text Manipulation Language file (HTML) to generate a web page, perhaps allowing the content to be varied by including some content generated from PHP variables or calculations.

  1. You will find advice on how to choose what language to use for scripts on PHP and Javascript page, along with references to some easy PHP scripts by Cumulus users.
  2. That advice continues with how to use a suite of PHP scripts to populate PHP variables with values taken from web tags at PHP Hypertext Preprocessor scripts reading web tags.

What file names are used when MX processes files

 

Cumulus use two slightly different processes depending on if the file is part of the supplied 'standard' web site, or an extra file you have specified on the Internet Settings|Files configuration dialog. For each of these processes there is an further variation depending on whether you have enabled the Use FTP rename option. This graphic shows the various combinations of file-names used for each step of the process. For the standard files, you can substitute any of the file names for indexT.htm, e.g. trendsT.htm

For more details of Cumulus Process see here.

I use web page packages provided by third parties

By third parties I mean items like those listed on page.

For most problems you probably need to contact the author via the support forum, but here are some possible answers

  1. You did not follow the instructions supplied for using that package, and you have used wrong file names or not put all the files where they should be or got confused between one-off transfers to your web site and files that Cumulus needs to upload at various intervals.
  2. File names in that package conflict with file names you already have on your server (this might stop any PHP require/includes linking in correct file)
  3. Maybe you are mixing some items using "http:" protocol and some using "https:", you need to chosse one or the other, not a mixture.
  4. Perhaps the package depends on what version of JavaScript, PHP, MySQL, you have installed, or on some setting for those languages. Depending on your web server, and the sort of access provide by any web provider you use , you might have a "cPanel" where these can be changed, or maybe you have to ask the provider to do the necessary changes.

Unsorted stuff

Some of the following is out of date, it refers to the original beta versions of MX not the latest release, other parts are just hints of what needs to be said ... ANY volunteers to UPDATE THIS?

It needs rewriting to be proper question and answer layout.

  • There is a third party offering for helping you start and stop Cumulus MX on a Raspberry Pi. There are two other related threads initiated by same author (Jank).
  • One problem various people have had is swapping from a Windows environment to a Linux environment. There are lots of new commands to learn, and specifying the interface is different so getting MX to connect to your weather station has been a problem for some.
  • For setting up your web site see this page first.
  • For a guide to the settings pages of MX user interface see here.
  • Cumulus MX supports updating to secure web sites, this feature is not yet properly documented.
  • The use of decimal comma gives rise to various problems mentioned in the forum, as the guidance above says Cumulus MX uses procedures that depend on full stops being used for the decimal point in many log files.
  • There is confusion between Cumulus 1 symbols for months (mm or MM allowed) and the MX use of those symbols (mm means minutes, only MM is valid for months). If you used Cumulus 1 and set up NOAA report names using 'mm' instead of 'MM', then you have a problem in MX. Another issue raised from time to time in the forum is that if you swap from Cumulus 1 to MX it may not access the old NOAA reports if the settings in MX are set to generate different file naming to that previously used in Cumulus 1).
  • Similar problems occur for those swapping from Cumulus 1 to MX if they have used any web tags that have had a date-time modifier added. Here not only do you need to be careful whether you are using the right case, but also be prepared for single modifiers on their own to have a different meaning to when that modifier appears with other modifiers.
  • On Cumulus 1, some people used '.' to separate the hours and minutes (HH.nn), this causes problems in MX that only accepts a colon separator (HH:mm). Thus files like the alltime.ini might have a mix of times in different formats.
  • Certain versions of Mono give problems with some locales, for example the Mono version 5 release has thrown up an issue with some locales and the short month names having an extra decimal point that Cumulus MX cannot cope with. Most of these problems are raised in the forum.
  • If you want to use your Cumulus 1 data with MX, and you use decimal commas, you will need to edit the .ini files to change the decimal commas into periods/full stops. The other data files will be OK.
  • If you are trying to use your Cumulus.ini file from Cumulus 1, you will need to make sure it doesn't contain any non-ASCII characters
  • Cumulus MX doesn't change the fact that you can't run two programs at the same time with the same station, whatever those programs are, unless you have something in the middle handling the multiple connections (e.g. Virtual VP for Davis stations)
  • If a settings screen shows red contents (red text and red boxes), invalid data is present which must be corrected before the form can be sent. One or more items on the page will have an error message.
  • Cumulus MX doesn't create any image files
  • Cumulus MX on Windows requires .NET 4.5 and hence will not run on the obsolete Windows XP
  • The realtime interval on the internet settings screen is in milliseconds
  • There is currently no 'diary' and hence also no snow recording
  • The requirement to run as root/administrator may possibly be removed at some point
  • When you install a new version of Cumulus MX (i.e when upgrading from one version to another), be sure to copy all of the files from the release zip file into your MX installation, and force a refresh of the interface pages in your browser. You can optionally just copy the files that I name as having changed in the announcement, but the risk is that I might have missed a changed file from the list. No settings or data are overwritten by the update.
  • Some setting changes don't take effect until you restart Cumulus MX
  • Note that the change to date/time formats in webtags also applies to NOAA report file formats., if you carry over your old settings
  • The 'extra files' editor updates automatically when a field is changed. You can press 'enter' in a text field to get it to update, or move to another field
  • The remote filename for an 'extra file' must include the path (if required) as well as the file name, just as in Cumulus 1. It does not use the 'directory' setting automatically
  • If you are using your 'old' Cumulus.ini file, make sure your 'ftp site' section is [FTP site] with 'site' in lower case
  • Davis stations sometime supply 'error' values such as 255 for wind speed. Cumulus ignores these and may log some of them to the diags file, if you have debug logging turned on
  • Serial device names on Linux are not COM0 etc as in Windows. You need to specify something like /dev/ttyUSB0 - do a dmesg at a command line prompt and look for the actual device name from your adapter. If dmesg says your cp210x converter (as on Davis stations) is attached to ttyUSB0, for example, you need to put /dev/ttyUSB0 for the serial port name in MX. You can set the serial port name in the MX user interface, but if for some reason you want to do it by editing Cumulus.ini, note that the item in MX is called ComportName (which is a string, for the full device name, e.g. COM1 or /dev/ttyUSB0) rather than 'Port' which is just an integer. If you have not run MX yet, this item will not exist, so you will need to add it (in the [Station] section).
  • If you are having problems after upgrading to Windows 10, see this post - https://cumulus.hosiene.co.uk/viewtopic.php?f=2&t=14148
  • Cumulus MX will not run on Windows IOT
  • The Dashboard, Now, and Gauges pages all get their data using web sockets (on port 8002). If they are not displaying data for you, you have some issue somewhere on your system preventing that from working - a firewall issue, or perhaps a browser issue. If you can't get web sockets working on your system, you could switch to using Ajax for the data. See the announcement for build 3032 for instructions.
  • If you want to use a twitter.txt file and include any characters not in the 7-bit ASCII set (e.g. degree symbols or accented characters) you need to save the file as UTF-8 (without 'BOM')
  • As with Cumulus 1, if you have extra web files configured, only tick 'process' for those files which have web tags in them, i.e. they need processing.
  • An "error [object Object]" in your browser when you try to change settings is usually caused by using a version of Mono which does not support comma decimals, but can also be caused by an invalid setting
  • Twitter have changed their authentication method, and Cumulus is now unable to perform the initial step of obtained the authentication tokens. The fix requires significant change to the code and I have no forecast for this
Retrieved from "https://www.cumuluswiki.org/index.php?title=Cumulus_MX_FAQ&oldid=5797"