5,838
edits
Migrating from Cumulus 1 to MX: Difference between revisions
m
no edit summary
m (Update category) |
mNo edit summary |
||
|
Like other longer pages in this Wiki, the content is divided into sections, if you are interested in a particular issue, find it in the table of contents, rather than reading whole page!
=Introduction=
The legacy Cumulus will never be updated, and as more people migrate to MX there will be fewer people to help you if the legacy cumulus software gives you a problem. MX has a major advantage it runs on more devices, and for example a Raspberry Pi computer is far cheaper to leave on all the time to run MX (knowing it will never be rebooted by Windows updates). The current developer of MX offers support, and other people are also gaining expertise, so if you have a problem you will get help.
=How to install MX=
The basic instructions for installing MX (see [[MX on Linux]], [[MX on Windows OS]], or [[Raspberry Pi Image]], as appropriate for fuller instructions):
#Download the MX distribution (from [[Software]] for latest version or from https://github.com/cumulusmx/CumulusMX/releases if you want an earlier release version)
#Unzip into the location where you are going to install MX.
== 'Engine' and 'Admin Interface' for MX ==
When you run the original Cumulus software it displays a screen (see [[Cumulus_Screenshots#Main_Screen]], and from there you navigate to other screens to view data or change settings.
MX is different, it consists of a stand-alone 'engine' which performs the reading and logging of data, uploading to a web site etc. This 'engine' is a command-line/terminal/console application which has no user interface. It does write diagnostic information to a [[MXdiags_folder|diagnostic log]], but many people run MX on a device that has no monitor and so the terminal output (if any) is not monitored.
When you successfully start MX, the engine is running, and it continues, until it is terminated by control C (or its equivalent in a Mac environment). (You can also run MX as a service, that has different ways to start and stop, discussed in the links given in previous section).
The separate [[MX Administrative Interface|admin interface]](unfortunately this is called various names in the support forum, including "user interface", "dashboard interface", and "admin interface") is provided by virtue of the engine acting as a web server. You can view the admin interface by typing the URL of the built-in web server into your browser, either on the same machine, or on a separate machine sharing the same local network.
If you install MX on Microsoft Windows, then a few extra one-off steps are needed to allow this web server functionality:
# Windows Defender has to be told to allow all for Cumulus MX.
# Typically for other security packages, select "CumulusMX.exe" then right click and select "Change File Rating to Trusted"
# In any "Firewall" package, add port 8998 as one that was permitted.
# Using "command prompt as administrator" type <tt>netsh http add urlacl url=http://http://192.168.1.64:8998/ user=\users</tt>, the response "URL reservation successfully added" means it worked. This command is apparently to allow all users to bind to port 8998 (i.e. that used for the Cumulus interface). This also means you don't have to run the engine (CumulusMX.exe) in an administrator user, nor select "Run as administrator" from right click menu on the shortcut, nor set the properties for any shortcut to run in any special way.
The default URL if the browser is on the same machine as MX is http://localhost:8998/. Newer releases of MX will tell you an IPv4 address to use, such as by typing "http://192.168.1.64:8998/".
Equally, if "localhost" is already in use for another web server (that you already run on your device), unless you can differentiate purely by port number you may need to use the correct IPv4 address as above, even on the same device.
For security reasons, the admin interface should not be accessible via the public internet.
=Migrating existing data files from legacy Cumulus to a recent build of MX=
If you want to continue using the same weather station, and you are not moving to a different home, you will want to maintain the data you collected using Cumulus 1 in your MX installation. This is what is meant by migration, and is the subject of the rest of this article.
==File names==
Note that if you run MX on a UNIX based operating system (e.g. Linux or Raspberry Pi OS) all file names are case-sensitive, please read documentation to see where capital letters are required in those file names. Be aware that Wiki pages change first letter to a capital even when a file that must be all lower-case is being described.
==Line terminators==
If you are not running MX on a Microsoft Windows computer, ideally you should change the characters appearing at the end of every line in any file you move from Cumulus 1.
I say ideally, because although Microsoft is fussy, and determined to be different by insisting files match its choice, most other operating systems can tolerate different line terminators.
With Cumulus 1 (or MX) on Windows, each line in every file ends with both carriage return and line feed.
With MX on a Mac, each line should end with just a Carriage Return.
For all Unix-based operating systems (Linux, Raspberry Pi OS, and other variants), each line should end with just a Line Feed.
To change end of line characters, run each file through an editor designed for programmers. Various editors are available but "Notepad ++" is one that is popular on Windows, but can run on other operating systems. In its '''Edit''' menu, choose '''EOL conversion'''.
If you use PHP Hypertest Pre-processor anywhere on your installation, normally that is written for "Line-feed" as line terminator and any "carriage return" in your files may mess up the content unless you code in a "trim" to remove it. Hopefully, if you use PHP, you are technical enough to realise you may need to edit the code depending on which device it is being run on!
==Configuration file==
MX's [[Cumulus.ini]] has different content to the legacy [[Cumulus.ini_(Cumulus_1)|'''cumulus.ini''']].
Cumulus 1 can recognise, in some circumstances, "cumulus1.ini", and other variants, not just "
MX only recognises "Cumulus.ini".
The old approach for migration, was to copy across your existing file, and let MX ignore all the parameters that do not apply to it. To add the parameters that MX does need, you would then go to the [[MX_Administrative_Interface#Changing_Settings]] pages and work steadily through ALL the options. For any parameters that were not set by the admin interface (and there were many "read-only" parameters in early releases of MX), one assumed these also existed in Cumulus 1 and so were already in your file.
At releases like 3.3.0 and 3.6.0, amongst others, there were changes to "read-only" parameters for MX. These could not be adjusted through the admin interface, and needed to be entered manually as listed on [[Cumulus.ini
Substantial changes to changes were made from release 3.8.0 onwards, hence [[Cumulus.ini]] is a new page applicable from that release. More changes were made in 3.10.0, and subsequent releases, as settings that used to have to be made directly in file were gradually moved to be advanced settings controlled in the admin interface. From release 3.12.0,many more new settings were added, more settings were removed, and some parameters in the file were renamed.
Consequently, if you are migrating from the legacy software to MX now, it is best to rename your old '''cumulus.ini'' file so it is is not seen by MX. Let MX create a new configuration file with just the parameters it needs using the various [MX_Administrative_Interface#Changing_Settings|settings pages in the admin interface]]. That will ensure you don't get muddled by parameters used for Cumulus 1 (but not for MX); and you do have all the parameters you do need, set correctly.
==Settings==
The settings pages in Cumulus 1 and MX work differently:
* for Cumulus 1 you choose to save changes by clicking OK,
* for MX changes are only saved when you click a '''Save''' button if one is provided.
* If there is no Save button anywhere on the screen (as in Extra Web Files) then the setting is saved when you move to next field/line.
Be aware that you should restart MX after changing settings, as many settings are only read from file when MX starts.
===Start date===
From release 3.10.1, MX allows you to edit this start date within the admin interface. It is "hidden" as an "advanced" setting, with a strict danger warning, but it is there!
===
You can skip this sub-section if your weather station connects to mX either by uSB or by wireless connection.
If your weather station used a port to connect to Cumulus 1, that port was set on the settings screen as a number and stored in Cumulus.ini_(Cumulus_1) in the station section as a parameter in the format '''Port=n'''.
If your old parameter had a value of '''3''', and you are still using Windows, the new setting would have value of '''COM3'''.
A typical parameter value for other serial connecting devices might be "/dev/ttyUSB0".
===RG11 Rain gauge===
==Strings.ini==
This is another configuration file, but it is an optional one.
If you install MX on another operating system, then the file system is case sensitive, in this case MX will only recognise "strings.ini".
If you have not created a [[Strings.ini|strings.ini]] file in your (leagacy) Cumulus top level folder, then you have no file to move to your MX installation, and you should skip the rest of this sub-section.
The contents of the [[Samplestring.ini|samplestring.ini]] file you get in your MX release distribution varies depending on the release you have downloaded.
Check your existing '''strings.ini''' file against the ''samplestring.ini'' file in the MX distribution you have. If the attribute names (left hand side of the equals sign) match for the parameters you selected to include in your '''strings.ini''', then you can reuse your existing file. If your file includes attributes that are no longer in the MX ''samplesting.ini'' file, then you will need to edit your '''strings.ini''' file that is placed in the folder containing CumulusMX.exe.
==NOAA style reports==
=="data" folder==
You should copy all files in the [[data folder]] from your legacy Cumulus installation to your MX installation if you want to be able to see past data.
There are some complications:
* See [[#Start date]] earlier on this page for a change that you may need to make to ensure MX accepts data in any old [[Standard log files]] and [[Extra Sensor Files]]
* See [[#Weather Diary]] later on this page, and [[Weather Diary]] page; Cumulus does not provide any utility to convert the old legacy format to the new MX format.
* See [[#Extreme Record files]] later on this page, and [[:Category:Ini Files]], for information on differences between the legacy files and the MX files
* See [[#dayfile.txt]] later still on this page, and [[Amending dayfile]] page, for how you can ensure MX can read your existing dayfile.txt file
* See [[#Line terminators]] earlier on this page for some technical issues if MX is running on a different computer to your Cumulus 1 installation.
===Weather Diary===
Please also note that the Cumulus 1 weather diary permitted multiple entries to be stored for an individual day, but the MX implementation only permits a single entry per day. Also the configuration file defines a time when the entries stop applying that may not match the rollover time for other weather measurements.
==Extreme Record files==
The [[Correcting_Extremes]] page explains how extreme records are stored in the [[:Category:Ini Files]] discussed below, with the changes to all-time records being tracked in [[Alltimelog.txt]].
You will find that MX creates at least one file that was not in the older Cumulus ([[Correcting_Extremes#How_do_I_correct_my_monthly_all-time_records.3F|monthlyalltimelog.txt]]).
===the .ini files===
Your Cumulus 1 '''.ini''' files can be read in all release versions from 3.0.0 to 3.5.y. Steve Loft did make two changes in his beta MX, these mean your .ini files may look strange while they have some entries made by Cumulus 1 and some made by MX:
# all newly stored values will use decimal points in these files, (i.e. any decimal commas valid in the legacy software, are not used by MX).
# all newly stored time-stamps will use the ISO specification (ISO 8601 Data elements and interchange formats – Information interchange – Representation of dates and times) so new dates have year first, the parts of the date are separated by hyphens, and new times use colon (:) between hour and minutes (i.e. the regional settings that the legacy software adopted have been abandoned by MX)
* '''MX releases from version 3.0.0 to 3.5.4''' can read all the date/time entries in any file whether in the Cumulus 1 format or in the MX format.
* MX will change any dates to the ISO format only when it needs to update that particular date/time.
* Consequently, within a single file both formats will co-exist, you may see lines using Cumulus 1 format for the extremes that have existed for a while, and Cumulus MX format for any new extremes.
If you used decimal commas in your Cumulus 1 installation, you will make the migration to MX easier, by an edit of each ".ini" file to change those commas to full stops.
Equally, if you had any times not using a ":" as separator, you will make the migration to MX easier, by an edit of each ".ini" file to change those characters into colons.
====MX releases at versions 3.6.x onwards====
Migration from the legacy Cumulus to MX has become harder as MX has been developed for two reasons:
# It appears that releases at versions 3.6.x onwards have become more fussy about existing content in these files due to changes in the source code.
#* Effectively, the format of existing entries is expected to be same as any newly created entries, i.e. matching current locale settings
# The addition of Feels Like temperature and Canadian Humidity Index (Humidex) to the .ini files
#* Neither of these was stored by the legacy software or by earlier MX releases
#* The technique for adding this is described on [[Calculate Missing Values]] page
Despite this, most people migrating from Cumulus 1 to the newer MX releases, are doing so successfully, so it works for at least some old files.
====
When Steve Loft designed his original Cumulus (1), he had no experience to draw upon as to the best way to treat items like dates. He wrote the software originally just for his own benefit and did not need to worry about time zones. Subsequently as he enhanced his software to make it usable by others, he faced many issues on how to cope with different time zones, and different weather stations having different sensors. In Cumulus 1, he basically focussed on compatibility by keeping to his original design for the data log files (both those ending in '''.ini''' and those ending in '''.txt''', and only adding extra fields to the end.
When Steve Loft took a new look at the data log files for Cumulus 2, he started with a new design, the principal change was that he decided to use UTC for all fields in them that reference dates and times. Steve struggled with Cumulus 2 largely because he was (at that time) not familiar with the C# language he was later to use for MX. It is fair to say that
When Steve Loft designed Cumulus MX, he was able to learn from his experiences with both Cumulus 1 and Cumulus 2,
===dayfile.txt===
MX (except early releases) reads the whole of this file as it starts running, so all lines must use exactly the same format.
====date format issue====
For MX (from release 3.5.4 onwards), the date separator specified for the locale when you run MX must be used in all lines of this file. Please see [[Amending_dayfile#Date_Separator_in_MX]] for more detail of what MX expects.
Cumulus 1 allows the date that appears in the first field of each line to use any character (except space) to separate the day of the month, from the month, and from the last two digits of the year. Consequently "29/03/88", "29-03-88", and "29.03.88" are all acceptable in the legacy software (and indeed in some early MX releases).
*See [[MX_on_Windows_OS#Parameter_for_changing_Locale]] for how to specify locale if you are running on Microsoft Windows (by default the locale settings are taken from the standard Windows settings application or Control Panel).
*If you are running MX in a Linux operating system, the locale parameter can be used, but the default locale is determined by the mono-complete package, and that in turn depends on your device settings.
So when you migrate "dayfile.txt", run it through an editor designed for programmers. Many such editors (e.g. Geary, NoteTab Light, Brackets) are available.
For example "Notepad ++" is one that is popular. That editor has a "Search" menu, with a '''Replace...''' option within it. In that option, there is a "Find what" box, a "Replace with" box, and a "Regular expression" selection.
Suppose your latest lines use either "." or "-" as separator, and that is what MX expects, but you have some older lines using "/" as date separator. Since you know "/" does not appear anywhere but in the dates with an older format, putting "/" in '''find what''' box and either "-" or "." (as required) in '''replace with''' box will sort you out after you hit "Replace all".
It is all a bit technical, if MX expects "/", but you have "-" in some older lines. The complication comes because "-" may be used in value fields too, so you need to find a way of specifying a minus with two digits before it and two digits after it. For this correction you need to select '''Regular expression''' and then set '''Find what''' to "^([\d]{2})-([\d]{2})-" and '''Replace with''' to "$1/$2/". Please check this, as this was copied from a forum post by Mark Crossley and I have not verified it.
====time-stamp format issue====
The dayfile.txt contains time-stamps following any value that represents a highest or lowest in the day.
All such time-stamps must be expressed quoting hour and minutes, with a separator. The use of seconds is not accepted.
The legacy Cumulus was not fussy on the character separating the minutes from the hour.
MX will only accept colon ":" separator, all dayfile.txt time-stamps must be in "HH:mm" format. You will need to edit your old lines if any use a different separator.
====value format issues=====
If you moved your Cumulus 1 installation from one windows pc to another, it is just possible that you might have a mix of "decimal comma" and "decimal point" in your values, or you might have changed the field separator (normally ";" or ","). Again, these must be consistent in all dayfile.txt lines for MX, and must match what is defined in the locale used.
=web server=
Some people have been slow to migrate from the legacy Cumulus to MX because any web pages designed using a [[Cumulus template file]] that works with the legacy software will not work with MX. It is not appropriate for this page to explain how to edit your [[Customised templates]], but it can briefly cover the different approaches for default web pages.
From release 3.10.1, the new default web pages have a totally new look (designed by Neil Thomas), and include responsive code allowing them to look better on wide screens and on narrow screens (such as those on smart mobile phones). MX creates a number of [[:Category:JSON Files]] that are used to convey the data from your local MX installation to the web pages installed on your web server. Mark Crossley's [[SteelSeries Gauges|Steel Series]] is used for the MX gauges page. A single "use default web pages" setting sets up the settings required, see [[New Default Web Site Information]] page.
Steve Loft's Cumulus 1 came with a set of example template files (designed by Beth Loft) that could generate web pages to upload to your web server. It also produced a set of images representing standard graphs that could be uploaded to your web server and shown on the default "trends.htm" page, a set of images representing wind speed and direction that could be shown on the default "gauges.htm" page these were based on'''Web Dashboard Components for FreeWX and FreeWX-Wi''' (no longer available) and mixed images generated by Cumulus with plots drawn using JavaScript routines (before "Canvas" functionality)., and a moon image that was shown on one web page.
=Library software=
Any non-technical person can skip this sub-section!
MX uses a number of library software utilities like Highstock, jQuery, boot strap, and others, see [[MX_Basic_info#Library_software_for_your_web_server]]. Be aware that MX determines the versions of these it seeks, and they may not match the versions needed for anything on your web server that is not supplied in the MX release distribution. A browser tries to reuse components that are already loaded, so there is a possibility of the wrong version being loaded.
| |||