Your Own Server

From Cumulus Wiki

Crystal Clear info.png This document is 'Work In Progress' so content may not be complete or accurate!

Request to Readers

When everyone used Cumulus 1, the Cumulus Wiki was highly rated as being comprehensive (partly because there were multiple contributors).

MX development has been so rapid, with MX now having so much functionality the documentation has not caught up (partly because of lack of contributors).

Can you contribute? Will you make this page more useful by bringing content up-to-date?

This page describes how I set up a web server and a database server on my Raspberry Pi computer.

For more information about running MX in a Linux environment please see MX on Linux.

For more information about Raspberry Pi computers see Raspberry Pi computer page.

For more information on log files that you might want to convert to database tables, please see MX Weather Diary, Legacy Weather Diary, daily summary log (dayfile.txt), Speciallog.txt, Standard log files, Extra Sensor Files, and Air Link Log.txt.

Introduction

This page records the experience of setting up a web server and database server on my computer.

Most web servers run on Linux, although you can still install them on a Microsoft Windows PC.

It is important to say that many different web servers exist, I have chosen to mention a few, and I have tried various ones. Since “apache2” is very well known, and proabably the most widely used server, I will mostly focus on that. Sometimes, you might choose a simple server that is what is described as light, it uses the minimum resources, but also probably offers the least functionality. In contrast, you might choose apache, because it offers the most functionality, and because it is used so widely there are many people who can help you.

Web server and database server built into Cumulus MX

Web Server built into Cumulus MX

The Admin Interface is a set of web pages that allow you to look at your weather data and change settings.

The MX code will create the necessary web server, it uses the Internet Protocol (ip) address of the computer that is running MX and a port (:8998 by default, but MX has an optional parameter to make it use a different port). If you are viewing the interface on the same computer as runs MX, “localhost” can replace the ip address.

As an aside, there are two different ways to specify ip addresses (IPv4 and IPv6), but this is not place to explain the difference.

Databases built into Cumulus MX

You do not need to know what is in these databases to use MX. I mention them here to explain MX has a built in database server.

Cumulus MX includes two SQLite databases;

  1. The first database was added by Steve Loft, but he never documented what it is used for (see Cumulusmx.db) and in the support forum when someone asked, nobody was able to answer.
    • It still remains in MX, so it must be used for something. One could hazard a guess that as MX needs to maintain a lot of numbers (the extremes it monitors, the numbers it holds for the admin interface, the numbers it can output for recent graphs and historic graphs amongst others) this might be why it needs a database
    • If you do find out what the first database is used for, please update the Wiki article on it!
  2. The second database is the Weather Diary documented here added by Mark Crossley. Also see in the support forum for a topic comparing the differences between Cumulus 1 and Cumulus MX weather diaries.
    • The Admin Interface provides a page to view, or edit, a single date in the weather diary. You can install another tool that edits SQLite databases if you want (see next section).

The legacy Cumulus software used a XML file to store the weather diary.

Two ways for editing the MX Weather Diary

You can install phpLiteAdmin (the significant part of that name is in the middle) to read (you can display the whole table very easily) and edit (via a friendly GUI) either, or both, of the SQLite databases that MX uses as mentioned above.

Install ‘’’phpLiteAdmin’’’ using sudo apt install phpliteadmin. Please see MX_on_Linux#The_various_components_to_commands_for_installation for an explanation of that instruction.

On a Linux computer, the main script for running ‘’’PhpLiteAdmin’’’ is located at /usr/share/phpliteadmin/phpliteadmin.php. Also in the /usr/share/phpliteadmin/’’’ folder, there is a link to ‘’’phpliteadmin.config.php’’’ which is actually in the ‘’’/var/lib/phpliteadmin/’’’ folder.

By default ‘’phpLiteAdmin’’ expects any database it accesses to be in a single folder, and to have a file extension ‘’’.lite’’’. ‘’’phpliteadmin.config.php’’’ defines the folder where it expects to find the database as in ‘’’/var/lib/phpliteadmin/’’’ folder by default. ‘’’ phpliteadmin.php’’’ defines the folder where it expects to find the database as in /usr/share/phpliteadmin/’’’ folder by default.

This means you will need to follow the instructions in that main script to change the password, and to define the path to the database you want to read, as well as telling it the file name extension MX uses (both databases have the wrong extension to be found automatically).

You may need to change file permissions, and you may decide to use the symbolic link example given here] so MX accesses the diary at a location that suits phpLiteAdmin.

I find the admin interface is not always reliable for updating the weather diary, MX uses a very simple way to access hardware;

  • when a successful insert or update happens, a confirmation message is returned
  • when it fails to store an update, there is no error either on the editing screen or in the MXDiags file.

The admin interface has a calendar type interface to pick the entry you want to insert or update; this is probably quickest for any edits in current month. The phpliteadmin graphical interface will list the whole database table and let you edit (or delete) any row, you can also insert (to create a new row), or import/export (using SQL), or copy the whole database.

Note that sqlite databases use sequential file structure and so for updates rows are physically identifed by row number (order of creation), not by the primary key that most SQL accesses use.

I can use phpliteadmin, or the admin interface editor, for making changes to my Weather Diary database now. Both can be accessed from a browser on any of my devices.

Adding a web server and a database server for your own purposes

MX has (optional) functionality that can upload to a web server, it also has (optional) functionality that can update one or more database tables on a database server.

Frequently, people will use an external web server. This might be one of the Wunderground, Windy, Awekas, WeatherCloud, PWSweathe, WOW, APRS, OpenWeatherMap, and WindGuru external sites that MX can automatically update to. It might be you have subscribed to a commercial web server (and optionally database server). that an external provider supplies to you to run either the 3.10.1 onwards default web site, or it might be for pages you or a third party have designed.

If you have a computer that is left running all the time without consuming a lot of electricity), you might want to add a web server and database server to that device. For example, see Raspberry_Pi_computer_page for a computer with low power consumption that a number of Cumulus users have used for their web server and database server.

As with a commercial web server, you can use your own web server to run the default web pages available in the MX download package. Alternatively, you might want this option, if you are creating your own web pages, and want to try them out without exposure to the public over the internet.

You might select this option to save the subscription costs of a commercial web server (and optionally database server).

All in one versus separate components

To get maximum benefit when using with MX, you need multiple components:

  1. web server – apache2 is just one ofvarious options listed below
  2. PHP script - from [1]
  3. Database server - MySQL Community edition from [mysql.com] or MariaSQL from [2].
  4. Integrating components like ‘’’phpMyAdmin’’ (there are alternatives) from [3] which combines PHP and MySQL

There are a number of packages that combine all of these:

  • If designed for Microsoft Windows and including Apache2, PHP, and either MariaSQL or MySQL, the package is often described as ‘’’WAMP’’’
  • If designed for UNIX (including Linux), then XAMP will include Apache2, PHP, and either MariaSQL or MySQL

Obviously, there are other web servers, some come on their own, others come as a package.

I will give a few examples, but there are many more, each has its supporters, but also each has people who criticise it. You can search on-line and find both all the (good and bad) comments, and clear instructions.

This page will only describe one installation (on a Raspberry Pi) in any detail, otherwise it will become far too complex.

Install Apache 2 (or another web server)

I will first mention 3 possible web servers than can run on a Microsoft Windows personal computer (or server box):

  1. You might choose Apache 2 as it is probably the most comprehensive, so if you have enough space on your PC, install it following instructions at the official site [4] (or one of many other web sites that give instructions for installing apache2 on Microsoft Windows).
  2. You might choose ‘’’Uniform Server’’’, which is extremely simple as it comes as a package including PHP script, and MariaSQL database in one package from [5], although you may need to update to a newer release of PHP from [6] and a newer phpMyAdmin from [7] as Uniform Server tends to issue upgrades very rarely.
  3. You might choose to install the Abyss web server from [aprelium.com]

If you get components separately, then there is extra work to get them to work together, but you will find clear explantions on download sites. Generally, if you have an all-in-one package, it seems easier; and normally upgrading one component is also easy (providing its a minor version upgrade, it might be more complex with a major version upgrade).

I will next mention 3 possible web servers than can run on a Linux computer:

  1. You might choose Apache 2 as it is probably the most comprehensive, so if you have enough space, install it using sudo apt install apache2 -y.
    • You might want to add Fast CGI to your apache server (this is a way of interfacing interactive programmes with a web server, but if you don’t have any idea what that might mean, skip this)
    • To add this use sudo apt install libapache2-mod-fcgid.
  2. You might choose nginx, as it is a quite popular web server for small computers (like the RPi), install this web server with sudo apt-get install nginx -y.
  3. You might choose lighttpd, as a web server that is designed to use as little space on your Linux computer as possible, install it with sudo apt-get install lighttpd -y.

For a Raspberry Pi, installing two of these is covered at [8]

Install PHP Hypertext Pre-processor

As already mentioned, some web servers come in a package that includes this script language. Again, as mentioned before, such packages may be rarely upgraded so the included PHP might be an old version with bugs or security concerns.

Whether you are installing PHP because you don’t have it, or because you want to upgrade it:

  • On Microsoft windows down load a zip from from [9]. You will want a “multi-threaded” version to use with a web server, as several PHP web pages may need to run at the same time.
  • On Linux, the simplest instruction to install it is sudo apt install php -y, although in practice this will not install all the PHP modules you actually need
    • A warning, which version of PHP that command gives you depends on which php version is found first in the respositories your computer has in its source list, so you may end up with the oldest!
  • A better variarion is to actually list the version we want:

sudo apt install php7.4

    • This still does not give us the modules we want, but it does control which version is obtained
  • A still better variation is to list the components we want:

sudo apt install libapache2-mod-php7.4 php7.4 php7.4-cli php7.4-common php7.4-json php7.4-xml php7.4-opcache php7.4-readline php7.4-mbstring php7.4mysql php7.4-curl php7.4-gd pdp7.4-psqlite php7.4-pdo -y

    • ”libapache2-mod-php7.4” is just part of what integrates apache 2 and PHP
    • “php7.4-cli” is what allows us to run scripts in batch
    • “php7.4-common” and “php7.4-mbstring” are pre-requisites for various other modules, and the second is used by ‘’’phpMyAdmin’’’ and other tools
    • “php7.4-json” (see here and Category:JSON_Files#Data_Transfer_Format and “php7.4-xml” (see this section) are used to handle particular file types
    • “php7.4mysql” integrates php and MariaSQL/MySQL
    • “php7.4-curl” “php7.4-gd” are used by some third-party routines designed by Cumulus contributors
  • You can add another repository that generally has fairly new versions (a few weeks behind the official [10] site) using

sudo wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/php.list sudo apt update

    • With this repository, we can install the equivalents in PHP8.0, here they are:

sudo apt install -y php8.0-common php8.0-cli libapache2-mod-php8.0 php8.0-json php8.0-xml php8.0-readline php8.0-mbstring php8.0mysql php8.0-curl php8.0-gd pdp8.0-psqlite php8.0-pdo -y

  • The Linux package manager will ensure that any components, that are required by dependencies, but not listed specifically, are also installed
    • Loading from the "packages.sury.org/php" repository has one disadvantage, the dependencies there don't integrate the PHP with your web server;
    • installing a web server and PHP from the default repositories will pick up joint dependencies and thus integrate the web server and php.
  • PHP is not the only script language available, but it is quite comprehensive
    • being able to be used either in a fairly simple way by non-technical people
    • or in an object-oriented way for those more technical to achieve success with more complex scripts.
  • To check which PHP modules have been installed by the above command, type php -m.
  • If you later want to use a database (and a tool like PhpMyAdmin), then your php modules loaded must include at least mysqli and mbstring.


To test that php is installed, type php -v into terminal, and you will see the exact version that has been installed, a confirmation that it offers command line interface (cli), and a copyright notice.

Creating a Home web page on your web server

You may wish to create a index.php web page at (Linux Apache server) /var/www/html which is the web server root for browsing, or of course you may wish to copy or FTP here all your existing web pages. To view a php web page, go into a browser and type in a url with the same IPv4 address as you use for the admin interface, omit the port (:8998) and instead type in your web page name (e.g. //192.168.1.xy/index.php, where you need to determine digits that replace xy).

Finding your PHP configuration file

For sake of simplicity in this article, from now on I will assume the web server you installed is "apache2", change that segment (in the paths quoted below here), if you installed a different web server.

The web server main php.ini is found at /etc/php/m.n/apache2/php.ini (where 'm.n' depends on which PHP version was installed, that was found out in last sub-section).

You may need to edit this file for example to specify where your include files are stored (if not in same folder as script with require/include). Replace "apache2" by "cli" in the path for the batch php.ini file that you may also need to edit.

To run a php script in batch, type php - f <file_name>. You can redirect the output by adding a greater than symbol and the destination file (i.e. > log_file) on the same terminal line.

If you want the MX external commands to run a PHP script for you, use something like "sh" as the program to run (i.e. run shell script); and in the parameters something like /home/pi/CumulusMX/MXutils/autoEOD.sh will run a script "autoEOD.sh" you have added to the "MXutils" folder. In that script, you put something like (assuming you have added a folder 'batch' with a subfolder 'log'):

#!/bin/sh
# This MX batch command file is initiated automatically by Cumulus MX software during last stage of processing the end of a meteorological day
echo "It stores feedback in log file CumulusMXDailyBatch (file name ends with day of month)"
sudo php -f the_path_and_filename_goes_here.php > /home/pi/CumulusMX/batch/log/DailyBatch_Day$(date +%d).log


Install Maria database

MySQL database software is controlled by Oracle and not made available for inclusion in Raspberry Pi repository.

MariaSQL is an alternative that has largely similar command syntax so is likely to work with code (like MX) designed to work with MySQL. Since the MX developer (Mark Crossley) actually uses Maria DB, we can have plenty of confidence it is suitable.

To install this database server, we type sudo apt install mariadb-server php-mysql -y.

Our database files will be stored at (Linux) /var/lib/mysql by default. Our MariaDB configuration is stored at /etc/mysql/mariadb.conf.d/50-server.cnf, and it is the datadir= entry that controls where the database files are stored.

Making your database secure

We need to assign passwords to control access to the database by typing into terminal sudo mysql_secure_installation. That brings up a screen:

  1. where we are asked to type current password for the root (as no password has yet been set, simply press Enter),
  2. next type Y to signify we are going to set a new password for root,
  3. next type in a new password that you will not forget,
  4. next as explained on the screen we are going to say whether users must select a user name as well as a password, type Y to Remove anonymous users
  5. next we have to decide whether we will only be logging into the database on our Pi (using Localhost) or we might be logging in remotely; type Y or N respectively, but if you choose N remember someone other than you might guess there is a root user and might guess the password you set,
  6. next we have another option of whether to retain or delete a test database, answer N or Y respectively, I would keep the test database for now as you can play with it and then remove it later,
  7. finally you type in another Y as that will Reload the privilege tables and ensure all is set up for your access to the database later.

The message, when the process successfully completes, is "Thanks for using MariDB".


Getting web and database servers ready for use

We need to create a user for PhpMyAdmin (or adMiner or whatever) to access our database and another for Cumulus to use to access the database tables. At the moment our database access has the single root@localhost user we created when we installed MariaDB. The initial password was set then, and we need to use it to get access to MariDB monitor where we can insert some SQL commands to create these two users.

PhpMyAdmin on first start up will ask for username (here I choose "admin") and password, thereafter it will use same log-in (by default you see log-in screen each time you restart or if it is left idle for a long time), let us create a user called 'admin' for it.

The database name (here I choose "cumulus" for the data base name), user name (here I choose "weather" for the user name), and password, must all match those set in MX using the MySQL_settings in the admin interface. You could of course use PhpMyAdmin (or AdMiner) to create additional user names, and to create the database, but I assume all is done in the following bit of SQL. Remember, the Windows operating system is not case sensitive, but all Linux based operating systems are case sensitive (so whatever pattern of capitals and lowercase you choose must be used every time. Also all names must start with a letter, can contain only letters or digits (no punctuation), and must not be a reserved word ("password" is a reserved word, so you cannot use that for a password, nor for a column name).

Obviously, these names might not be what you use, but you can amend commands below accordingly. For each line with SQL, it must end with a semicolon (;) as shown. After you press "Enter" key you will get a response saying "OK" if you have remembered the semi-colon. You can actually use "\G" or "\g" instead of a semi-colon, but here we will keep it simple and stick to semi-colon.

The SQL lines have a prompt of a greater than symbol (>) while the command lines have a prompt showing current path. Note that "identified by" is followed by a password enclosed in single quotes.

sudo mysql --user=root --password=InitialPassword
create user admin@localhost identified by 'PhpMyAdminPassword';
create user weather@localhost identified by 'MXPassword';
grant all privileges on *.* to admin@localhost;
grant all privileges on *.* to weather@localhost;
FLUSH PRIVILEGES;
create database cumulus;
exit;

As I type this, Cumulus MX has no exception handling if the username and password defined in the settings do not exist in the database, therefore in this situation it will crash out (with message press Enter to close).


Installing Adminer, or PhpMyAdmin

ExportMySQL.exe and CumulusMX.exe (see Cumulus_MX#Executables for details) both create SQL for updating tables in a MySQL database, such as the one our MariaDB software package we have installed can create.

However, there is nothing in the MX distribution that lets us back up and manipulate (e.g. delete rows with errors, or correct rogue numbers in a column) tables in this database. This option is about installing a package to do the tasks that go with operating a database.

PhpMyAdmin

PhpMyAdmin is one tool that can be used to manipulate your MySQL like database (that is the significance of the "My" bit in the middle of the tools's name).

You may like PhpMyAdmin as this offers:

  • a graphical approach (you see a table on screen and navigate to the row or cell you want to work on)
  • a SQL approach (you can try out any SQL here, before adding it to a script that you might use in a web page)
  • a selection approach (you select a database, then select a table, then select an action)

WINDOWS:

  1. Download from official download page
  2. You need to do some configuration (PLEASE EXPAND THIS).

LINUX:

  1. The version of phpmyadmin in the default repository is not compatible with PHP7.2 and above
  2. Visit the phpMyAdmin official download page and look for the .tar.gz URL and download it using wget.
  3. The following steps assume version 5.1.0, released Feb 2021 is to be downloaded. (If a later version is now available, make sure to change the commands below to match)
  4. Decide whether you just want English, or need another language
  5. Change to folder where we will extract our download cd /usr/share/
    • For English only, extract your download into the current folder using sudo tar xzf phpMyAdmin-5.1.0-english.tar.gz
    • For other languages, extract your download into a sub-folder of the current folder using sudo tar xzf phpMyAdmin-5.1.0-all-languages.tar.gz
  6. Create a folder to hold the download sudo mkdir /usr/share/phpmyadmin
  7. Now we will move the contents from the sub-folder created by the "tar" command into the folder we just created
    • For English only, the move command is sudo mv phpMyAdmin-5.1.0-english/* /usr/share/phpmyadmin
    • For other languages, the move command is sudo mv phpMyAdmin-5.1.0-all-languages/* /usr/share/phpmyadmin
  8. There is some configuration to do
    • First sudo nano /usr/share/phpmyadmin/libraries/vendor_config.php to edit (with nano) the vendor configuration
      • change the path shown in the existing line mentioning 'TEMP_DIR' so it becomes define('TEMP_DIR', '/var/lib/phpmyadmin/tmp/');
      • change the path shown in the existing line mentioning 'CONFIG_DIR' so it becomes define('CONFIG_DIR', '/etc/phpmyadmin/');
  9. The following sequence of commands will
    • add the mysqli module to our php install, sudo phpenmod mysqli
    • will restart apache, sudo service apache2 restart
    • will create a symbolic link for the phpadmin installation in the server web root so it can be seen (and used) in our browser, sudo ln -s /usr/share/phpmyadmin /var/www/html/phpmyadmin
    • will give the standard user (pi) ownership of the database files and the web pages: sudo chown -R pi:www-data /var/www/html/
  10. phpMyAdmin now has the files where it expects to see them, and should work
    • To log in requires a username (default "admin") and password (defined in your database set-up - see previous section)
    • The remaining configuration can be done within phpMyAdmin, follow cues on each screen as work through all configuration tabs.


If you have any difficulty, there are some instructions at https://devanswers.co/manually-upgrade-phpmyadmin/ and some at https://docs.phpmyadmin.net/en/latest/setup.html#quick-install


adminer

You may prefer to install a different tool, perhaps adminer that works using a drill down approach. A drill down approach is when you select the database, then select the table, then select the row, then select the column, then select the action. This logical step by step approach is a popular approach, but does not suit everyone.

Install this drill down package with sudo apt install adminer (I leave you to work out the commands needed after that).


Viewing web pages on our new web server

You can view any index.php or PHPMyAdmin web page in your browser by prefixing the address with your Pi IPv4 address e.g. a URL like http://192.168.1.xyz/phpmyadmin where xyz is 2 or 3 digits you look up as mentioned before where FTP was described. If your Pi has both Ethernet and wireless connection, there will be two different values of xyz for you to choose one from. On first run of PhpMyAdmin, as already mentioned, you will see a PHP MyAdmin log-on page where you type username and password we have just set up.

Populating your database tables on your Pi

Cumulus MX has functionality to update database tables at one of 3 intervals:

  1. real-time
  2. standard interval
  3. end of day

The database tables can use the column names in the schemas pre-defined by Cumulus MX or in a custom schema (where you specify the column names). The settings are all in MX_Administrative_Interface#MySQL_settings, so read that section to find out more.

You might have started using MX before you set up your database. There is a option in that part of the admin interface to create database tables (as required) for each of the 3 updating intervals. For example, the default name for the table updated at the standard interval is "monthly", but you could give it a name of "standard" or whatever you like.

The MX release distribution includes another executable. Type cd CumulusMX && sudo mono ExportMySQL.exe monthly to run the executable. The first parameter is "monthly" even if the table has a different name.

  1. if the table name is defined in the admin interface, and the table already exists in the database with the correct columns defined, then the executable will use that table.
  2. There is an optional second parameter that specifies the log file name to read.
    • if the second parameter is not specified, this executable will look at every log file (in "data" folder and with file name that starts with month, then has "log.txt"),
  3. for each line in the log file the executable will try to insert a row in the database table
    • the SQL syntax used is "insert ignore", so if the row for that log file line already exists in the table, it will skip onto next line of log file.

Similar instructions apply for the end of day table, although as there is only one daily summary log file, there is no optional second parameter, just type cd CumulusMX && sudo mono ExportMySQL.exe dayfile.

There is no way to use this executable for insert of past rows into the real-time table.

If you have been running Cumulus on another device previously and already have database tables, the next section explains how you can create SQL to export your old database and use that SQL to populate the new table.

Transferring database tables to your Pi

If you have been using Cumulus before (and already have a database) then you can use PhpMyAdmin on your old device to export out all the Cumulus tables as SQL in a zip file, FTP that zip file across to your Pi, then use PhpMyAdmin on your Pi to import that zip file.

Providing you selected the right options for what SQL you created in your export, the import will contain SQL to create the tables and to insert all the rows into each Cumulus table on your Pi. Please note that there is a limit of around 1000 rows that can be imported in one action, so for your bigger tables, you will only create the table once, but you will do several Replace row actions (export from old device, then import to Pi) each transferring just one thousand rows, until the whole table is on the Pi. You may prefer to use "ExportMySQL.exe" as described in previous section to recreate your bigger tables from the Cumulus log files.

You might want to also export/import the tables in the PhpMyAdmin database (as they contain your preferences for each of your tables) in a separate zip, although these might need some extra transformations, as they are specific to a particular version of the database server (and the old database server version may not match the MariaDB version on your Pi). Some PhpMyAdmin tables do change for different versions of the tool, so that too may make export/import of its tables more complicated.

Restarting Web Server

After all these installs, we need to restart Apache (so it loads the PHP and MariaDB), by typing sudo service apache2 restart or (if we just want the Apache configuration reloaded) by typing sudo systemctl reload apache2. Similar commands apply for other web servers.

We will need to restart Apache (or whichever web server we installed) any time we change our php.ini files, database passwords, and anything else that is checked when the web server starts.