View source for Preparing your Linux computer for MX

=Preparing your Linux computer for MX=
[[Category:Cumulus MX]]
This text originally appeared on the [[MX on Linux]] page.

It describes what is needed once off to set up your computer ready for use with MX.

The developer has created [[Software#Raspberry_Pi_Image|an image you can download]] for those prepared to run two computers (a RPi for actually running MX and another computer for all interactions with MX).  Using that will avoid you having to understand, and action this page.


==Introduction==
Linux is available based on a multitude of different kernels (the building block for the operating system), on a multitude of devices.

This page has been originated by a contributor using the Raspberry Pi Operating System (this is based on Debian, one of the Linux kernels). Be aware therefore that some instructions on this page are specific to a [[Raspberry Pi computer page|Raspberry Pi computer]] with its default operating system.

For other devices, the inclusion of the correct instructions is totally dependent on whether any contributor has edited this page to cover your device in the context of that section of this page. It is hoped that contributions to this page will be made by Cumulus users with a range of different devices so this page is useful to more people.

:Until somebody creates a separate page for Apple Mac computers (that could be a good idea, as there are some significant differences), this page is the best source of advice.

=Install starts here=

If your computer has the pre-installed image, or you have installed MX before, you can skip the sub-sections that follow on here and continue reading at [[#Cumulus packages]].

== Preparing for an install==

Before we do an install of a new package, we typically use this series of commands to ensure our computer is in the best state to work out dependencies of what we are about to install:

:<code>sudo apt update</code>

:<code>sudo apt -y full-upgrade</code>

:<code>sudo apt autoremove</code>

Each of those can be understood from information in [[#install|earlier sub-section]].

If you are installing onto a Pi zero, or similar slow computer, please ensure the size of the swapfile is as big as possible, as the mono-complete we will install is large. In linux, we type <code>free -m</code> to see our RAM size and our swapfile size.

To change swapfile size on the Raspberry Pi (''Can contributors add details for other Linux computers''):
# Edit a file <code>sudo nano /etc/dphys-swapfile</code>.  
# Move the cursor down line by line until it reaches '''CONF_SWAPSIZE=100'''.  That is showing that the swapfile is only 99 mb by default in the Raspberry Pi Operating System. 
# Now move the cursor to the 100, and change it to "512" which is enough for mono even on a Pi zero. 
# Next, stop, and restart, the relevant service using <code>sudo /etc/init.d/dphys-swapfile stop && sudo /etc/init.d/dphys-swapfile start</code>. 
# That should complete quickly, and we can type <code>free -m</code> again to see the 99 we saw previously has been replaced by 511.

The actual swapfile size to choose depends on the amount of RAM installed on your computer, if you want to know more (because there is some conflicting advice), then do an online search to discover the RAM size on your device and the advice on suitable swapfile size.

=== USB HID === 

There is one more prerequisite package for MX with some weather station types.

You may notice [[MX_Basic_info#HidSharp.dll|HidSharp.dll]] in same folder as executable, this is a cross-platform [https://www.nuget.org/packages/HidSharp/ Universal Serial Bus (USB) Human Interface Device (HID) library].  

To check your USB devices, type <code>sudo lsusb -t</code>.  

MX uses this library for weather stations  (like Fine Offset and USB connected Oregon Scientific models) that appear as a HID via a USB connection. To check if your weather station appears as a HID, download this [https://cumulus.hosiene.co.uk/download/file.php?id=11414 USB HID test package] and run it.

This HidSharp library calls a package ''libudev shared library'' file called ''libudev.so.1''. Most recent Linux distributions will already include the libudev shared library. On my Raspberry Pi, [[#install|typing]] <code>apt search libudev</code> showed ''libudev.so.1'' was already installed.  If it is missing, then [[#install]] it.

Optionally, read about [https://cumulus.hosiene.co.uk/viewtopic.php?f=39&t=14310&p=111593&hilit=libudev.so.1#p111593 Oregon Scientific] issues.

Optionally, read about [https://cumulus.hosiene.co.uk/viewtopic.php?p=107913#p107913 Fine Offset] issues, and the optional symbolic file link instructions (those apply if you only have an earlier libudev shared library version, and wish MX to use that).

== Checking if mono-complete is in Source List ==

The "source list" [[#The various components to commands for installation|mentioned later]] may not contain all the repositories we need for our installations.

Consequently, [[#The_various_components_to_commands_for_installation|type]] <code>apt search mono-complete</code>, to find out whether the mono package is available from one of the repositories already in our source list, and if the version available is compatible with MX (Release announcements for MX should specify which versions of Mono will work).

If mono-complete is not available (or only available in an older version incompatible with MX), then we have to add a new repository, and the one to add depends on which Linux kernel is used by our Operating System.

There are a number of sub-sections below, please check which applies to you, and only read that one.

===Add the Mono repository for a Raspberry Pi===

The two Mono repositories listed here are specific to the 2017 and 2019 releases (respectively) of the operating system for a Raspberry Pi computer.  These are taken from  [https://www.mono-project.com/download/stable/#download-lin-raspbian download-lin-raspbian].  If your Rasberry Pi is running the 2021 release (11 = Bullseye), or whatever is available when you are reading this page, please refer to that external link for more recent instructions.

# the first line (in each case) installs a certificate
# the echo line defines a repository to add to the sources list.
{| class="wikitable"
|-
! scope="col" style="width:450px; color:blue;padding:8px 14px;font-size:16px;" | Raspberry Operating System 9 (stretch) !! scope="col" style="width:450px; color:navy;padding:8px 14px;font-size:16px;" | Raspberry Operating System 10 (buster)
|-
| style="height:50px;padding:14px;font-size:16px;"| <nowiki>sudo apt install apt-transport-https dirmngr gnupg ca-certificates</nowiki> || style="padding: 0 14px;font-size: 16px;" |<nowiki>sudo apt install apt-transport-https dirmngr gnupg ca-certificates</nowiki>
|-
| style="height:100px; padding: 0 14px;font-size: 16px;" | <nowiki>sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF</nowiki> || style="padding: 0 14px;font-size: 16px;" |<nowiki>sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF</nowiki>
|-
| style="height:100px; padding: 0 14px;font-size:16px;"| <nowiki>echo "deb https://download.mono-project.com/repo/debian stable-raspbianstretch main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list</nowiki> || style="padding: 0 14px;font-size: 16px;" |<nowiki>echo "deb https://download.mono-project.com/repo/debian stable-raspbianbuster main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list</nowiki>
|-
| style="padding:8px 16px;font-size:16px;"|<nowiki>sudo apt update</nowiki> || style="padding:8px 16px;font-size: 16px;|<nowiki>sudo apt update</nowiki>
|}

===Add the Mono repository to Ubuntu, Debian, Fedora===

At time of writing, [https://www.mono-project.com/download/stable/#download-lin-ubuntu download-lin-ubuntu], shows the instructions for versions 16, 18, and 20 of Ubuntu.

Equally, [https://www.mono-project.com/download/stable/#download-lin-debian download-lin-debian], gives details for debian, and [https://www.mono-project.com/download/stable/#download-lin-fedora download-lin-fedora] for Fedora.

Others can be found by choosing other tabs on any of those links.

== Installing Mono instruction ==

Now we have the certificate needed, and the repository for mono-complete is added to our source list, we can do the actual install:
: <code>sudo apt install -y mono-complete</code>. 

* The “sudo”, “apt”, “install”, and “-y” have [[#The various components to commands for installation|already been explained]].
* The "mono-complete" is the package we want.

:Please note that if you just specify “mono”, you will get ‘’’mono-devel’’’ (the developer edition) that does not include all the components required by each of the Cumulus executables.

Please note that a particular MX build might specify it needs a particular version of Mono.  Hence, although normally you can upgrade any (CumulusMX.exe, ExportToMySQL.exe, CreateMissing.exe, proposed CreateRecords.exe) cumulus package without upgrading Mono, sometimes you will need to upgrade Mono, and that means following above install instructions again.


'''The latest versions of MX require Mono v6.8 or later (6.12 or later if you want to use email)'''.<br>
Please check the version installed by the above process, if it is an older version then you will have to install Mono directly from the Mono web site as outlined below...


The latest release of Mono, for a variety of Linux distributions, can always be downloaded from the mono project web site [https://www.mono-project.com/download/stable/#download-lin mono-project.com] (the macOS versions are here [https://www.mono-project.com/download/stable/#download-mac mono-project.com]):
# follow step 1 there, 
# but in step 2 replace ‘’’mono-devel‘’’ by ‘’’mono-complete’’’

The complete mono package includes a component (mono-xsp4) that creates a simple web server to run ASP.NET 4.0 applications.  MX does not need this, so type <code>sudo update-rc.d mono-xsp4 disable</code>, to stop it being used (without needing to do a <code>sudo apt remove</code> followed by the name of component you no longer want).

=Moving from Microsoft Windows to Linux=

Microsoft has previously had a deliberate policy to make everything on its computers different to the standards set by earlier computers, to try to keep people using its products, transfers between these systems is not easy.


==Terminology==

First of all, be aware that storage devices (flash memory, SSDs, etc.) formatted for Linux cannot be read by Microsoft Windows, although some will be formatted with a root partition that can be read on both.

Microsoft defines drives (indicated by letter followed by a colon). Linux has all files in a hierarchy of directories that always start with root denoted by "/" at the start.

A particular storage device can be divided into partitions, on Microsoft Windows these are assigned to different drives, in Linux the partitions are assigned to particular directories within the hardware category, but often mounted as a directory with a different path.  Care must be taken to refer to the path where the device is mounted for all file operations, but to the hardware path for device operations.

Microsoft talks about folders, while Linux talks about directories. Both of these define paths to particular files.

Linux sub-directories and files can look the same in some ways of listing the content of a particular place in the path. Linux starts file names with a period (.) for "hidden" files (those you choose whether to show or not in any directory listing), Windows uses a flag to mark files as hidden.

==Encoding==

Cumulus MX expects all files to be encoded using UTF-8.  This means a single byte is used to represent characters, that byte has 8 bits and can represent numbers, letters, and certain symbols, this covers everything used in standard characters in most languages. Microsoft Windows software (like "Notepad"), and Google Cloud software tend to add by default a "Byte Order Mark", a special code at start of files that indicates the order of bytes used when more than one byte represents a character. If you edit a Cumulus file in any software that adds a BOM, or you have set encoding to include BOM by default, MX will be unable to understand any externally edited files, and it is likely that your Linux computer (or the MONO that MX uses) is set up to expect standard utf-8 (without BOM).  For all [[:Category:Files with Comma Separated Values|files with Comma Separated Values]] the first field should be a date, but a BOM prefix on the first line of such a file pushes the date out of expected position!

===Line terminators in .txt files===

If you are a novice to computers, you can optionally skip this sub-section and the next, going directly to [[#File Names & Paths]]. 

For normal MX usage, line terminators do not matter, MX is now capable of detecting what line terminators are being used, and will ensure it sticks to the existing system.

If you are using a script language such as PHP, or you are using a third-party package with your MX, then these are written expecting particular line terminators and may stop working if run on a different computer to the one used for their development.  By default, PHP routines for reading files will normally assume terminator is Line Feed, so if a file contains a Carriage Return, an appropriate "trim()" function has to be used to remove unwanted characters.

If you are moving from Microsoft Windows to Mac OS, be aware that Microsoft ends each line with two characters (Carriage Return and Line Feed) while Unix/Linux ends each line with a single character (Carriage Return).  Cumulus can cope with both approaches for existing files, but will create new files correctly for Mac OS.

If you are moving from Microsoft Windows to Linux, be aware that Microsoft ends each line with two characters (Carriage Return and Line Feed) while Unix/Linux ends each line with a single character (Line Feed). Cumulus can cope with both approaches for existing files, but will create new files correctly for Linux.

If you run your Linux computer in a headless mode, accessing its files by a remote terminal session, be aware that the line terminator used by the remote computer may be applied to files affected by whatever command you do remotely. Equally, running a Cumulus executable (MX or one of the utilities) may create new files with the wrong end of line terminator. The latest releases of both "CumulusMX.exe" and "CreateMissing.exe" have been amended to create their new files with the same line terminators as are used by any existing files they read.  Older releases adopted whatever line terminator was used by the device from which the executable run instruction was issued.

===Reading Cumulus files using your JavaScript, your PHP scripts, and packages from third-parties===

If you have written a script to work for a Microsoft Windows device, it might need editing to still work in a Linux environment.  If you use a third-party provided package on your Linux computer, you may need to check if the author has coded it assuming it will be run on Microsoft Windows.

It is not appropriate for this Wiki page to go into full technical detail, but here is a quick summary of how unexpected line terminators can stop scripts working:
* If the script expects the Linux LF, and finds CR LF, then the final field of each CSV line has an invalid character in it, so your script will not be able to understand that field as a numerical value (or time-stamp).
* If the script expects the Mac OS CR only, and finds CR LF, then after the first line, each subsequent line starts with a “LF”. Most scripts will be processing [[Daily Summary|dayfile.txt]], [[Standard log files|MMMyylog.txt]], or [[Extra Sensor Files]], all of which have date as first field, and the "LF" prefix stops that field being seen as a date.  In the worse case scenario, it might mean the script sees empty lines between each valid line.
* If the script expects CR only, but just finds LF, the script will believe the whole file is just one line, and the fields before and after the LF will be treated as a single field (merging the last value of one line with the date that should be on next line) so your script will not be able to understand that field as a numerical value (or time-stamp).
* If the script was written for a Microsoft Windows environment, it will expect CR LF, but might be confused if the end of line is different

===Editing Cumulus files outside MX===

If you try to edit a file outside MX, the application you choose to use will normally adopt the line terminators used by the device on which you are using the application.

Any good quality editor should by default save the file with the same line terminator as it had before.

If you are using an editor designed for editing computer files, it should have an option to [[#Changing line terminators|choose which line terminator you need]].

===Changing line terminators===

Many editors designed for programmers (they might be described as providing a programming development environment) can change the line ending of every line (either while file is being edited or when file is saved). 

‘’’Geany’’’ is a programming development editor available for Microsoft Windows, but included by default on some Linux systems including Raspberry Pi, that can do this ('''Document''' menu, -->> '''Select Line Endings''').

''Notepad++'' is another editor available for multiple operating systems ('''Edit''' menu -->> '''EOL conversion''').  

Both have capabilities to make such changes on either the single file that has focus, or all loaded files.

There are many more editors designed for programmers, and you may use one that is not listed above, so you may need to search for the relevant option.


===File Names & Paths===

Another issue that may be encountered when moving from Windows to Linux is the difference in File Names & Paths.
All Unix file names are Case Sensitive, and the separator is "/".   

Microsoft Windows Operating Sytem deliberately chose to be different and uses "\".  That causes a problem because "\" is also used to denote a control character (e.g. '\t' is horizontal tab, '\n' is new line) and so Unix based computers will strip out any two character sequences they recognise as representing control characters from any file paths they read.

This could be applicable if you were using Extra Web files functionality in your old Cumulus installation on a Microsoft Windows computer.  You need to go into the settings page for  '''Extra web files''', and edit each local path for it to work on your Linux computer.  If you have followed advice on [[Customised templates]] page, you will be using a location outside the MX file structure, so you knew you had to change local file paths anyway for your new MX installation.

Here is an example changing the case and the separators where the path is within the MX file structure: ''Web\extrapageT.htm'' in Windows, changed to read '''web/extrapageT.htm''' in Linux.

=That is enough folks=

If you have read up to here, you now know the basics for using MX on Linux.

The remaining sections are more technical and so you can skip them.

=Technical Extra=

PLEASE SKIP ALL SUBSEQUENT SUB-SECTIONS IF YOU WANT TO AVOID TECHNICAL EXPLANATIONS.

== external storage ==

Generally, if you attach USB storage (a disc or a stick), Linux OS distributions will detect any existing partitions (yes a technical term) and allow you to read files stored in them. This applies whether the partition is formatted for Linux or for Microsoft Windows.

However, you may have a brand new, unformatted, drive, or you may want to delete, or add partitions, or to format them as Linux partitions (as these make the input/output significantly more efficient).

You can install software that uses a GUI to make this easy, e.g. '''gparted''' [https://gparted.org/ partition editor].


Alternatively, you can use a terminal session, and lots of commands:
#connect your external storage
#type '''su''' to gain administrative access
#enter your RPi password
#type '''fdisk -l''' (this is only available to root user) to see names for all storage your Linux computer can see
#an external drive will be named something like ''/dev/sd''''a''''' although that "a" might be "b" or a subsequent letter in alphabet depending on what has already been assigned
# if "sda" and "sdb" appear, or any others up to "sdz", the last one will relate to the most recently connected storage
#if your drive has partitions, then you will see further entries like '''/dev/sda1''' and ''/dev/sda2''.
#type '''df''' to see whether your drive is currently mounted (being used by computer system)
#if it is mounted, the command to use next is (type this accurately, there is a temptation to type an English word that adds an extra "n"!) <code>umount /dev/sda</code>, obviously replace the "a" by the appropriate letter seen in the earlier command
#if the drive does not have a partition, create one using <code>fdisk /dev/sda</code>, again changing the "a" into whatever letter was seen in response to the first "fdisk" command
#*"fdisk" is a utility, it will wait for further instructions, follow each with pressing "Enter"
#*type '''n''' as instruction to create a new partition
#*type '''p''' to make this the primary partition on this drive
#* type '''1''' to make this the first partition
#*accept default offered for first cylinder
#*accept default offered for last cylinder, if this is only partition, as that ensures the whole disk (apart from partition table) is available for your data
#*for simplicity, this guidance will not cover the possibility of multiple partitions
#*type '''t''' to say you are specifying the way you want this partition to be specified in partition table
#*optionally type '''L''' to see what file system types are available for the partition table
#*to select a "Linux" partition, type '''83'''
#*type '''w''' to create the partition you have now specified for Linux.
#Now we have a partition table and a partition on our drive, we can repeat '''fdisk -l''' to see the entry now added, it might be '''/dev/sda1''', where again the "a" might be a different letter
#To format this partition for Linux, we specify "ext4" as the way to format it using <code>mkfs.ext4 /dev/sda1</code>, again replacing the "a" as required.
#we need to create a folder within 
#*"/media" for Linux in general
#*"/media/pi" for Raspberry PI OS
# As we will learn later, the relevant command (in RPi OS) is '''mkdir /media/pi/my_short_name''', where "my_short_name" is selected by you
#To mount our partition, we type <code>mount /dev/sda1 /media/pi/my_short_name</code>, where "sda1" is replaced by "sdb1" or whatever we saw in '''fdisk -l''', and "pi/my_short_name" is replaced by whatever we used in our make directory command.
#To (optionally) get our partition mounted at boot, we can use an editor (see later) to change the boot instructions, by typing <code>nano /etx/fstab</code>
#*In the editor, use the down arrow on your keyboard to move to last line, and then type '''/dev/sda1''' (change the "a" as necessary), then press "tab", then type '''/media/pi/my_short_name''' (change "pi/my_short_name" to whatever we used in our make directory command), then press "tab", then type '''ext4''' (again matching the format type we  selected earlier), then press "tab", then type '''defaults''', then press "tab", then type '''1''', then press "tab", then finally type '''2'''
#*save the file (as described later in nano sub-section), hold down control key and press '''o'' letter key. Press "enter" again to confirm same file name.
#* exit nano by holding control key and pressing "x" key.

=Operating systems=

New hardware might come preloaded with an operating system, or might allow you to choose which operating system to install on it.

Please see [[Raspberry Pi computer page]] if you want guidance on choosing which model to buy and how to install an operating system on those computers, so you are ready to install MX.

=Interactive Package management=

All Linux computers include a package manager, there is some variation between them.


==Package tool==

You can add extra programs to your Linux computer.  Programs written to run in multiple operating systems are usually downloaded  as ''*.tar.gz'' files, although other file compressions may be used (.zip is effectively Windows specific, but as we shall see in [[MX on Linux]] can be used on Linux).  Programs written for Linux distributions like "Red Hat", "Fedora", and "centOS" are supplied as ''*.rpm'' files.

For the purposes of keeping this Wiki page simple, the package manager described here is the modern debian one “apt” meaning “Advance Package Tool” e.g. <code>sudo ''apt'' install package_name</code>. 

In simple terms, the inclusion of "apt" runs the “package manager” used in any modern Linux based on a Debian, or Ubuntu, distribution where programs are written in files ''*.deb''. That is certainly true if you use the Raspberry Pi operating system, and it appears it is true for almost all modern Linux operating systems. 

Some online tutorials refer to “apt-get” or "apt-cache" in examples they quote. My online research suggests using those directly is no longer advisable, and therefore you should leave off the part starting with a hyphen, i.e. it should be safe to replace these older package managers with "apt". I have included [[#Package Manager – a brief technical aside|a little technical explanation]] after this subsection, if this really worries you, it appears that in some Linux variants "apt" is just a more user friendly way to call the older package.

If you are using an older operating system, it appears the tool available depends on the variant of Unix you have installed. The contributor of this section has done some online research, but is not familiar with all the Unix variants, so other contributors with fuller technical expertise may need to edit this section to make its applicability wider.

According to online documentation seen, some UNIX variants use  '''RPM packets'''. The modern package manager for these is "dnf" and in general you can substitute "dnf" for "apt" in my descriptions. However, there is an older package manager for RPM packets called  "zypper", again in general you can replace "apt" in my notes with "zypper" and it should work e.g. <code>sudo zypper install package_name</code>.  Other package managers used with "*.rpm" files include 'yum' (standard with "Fedora" distributions) which does not use the same syntax as "apt", and 'rpm' (which can only install packages that are already on your computer but currently dormant) again using a different syntax.

== Package Manager – a brief technical aside==

Linux operating systems install software by looking in repositories, and checking a register showing dependencies.  When you ask Linux to install a particular package using “apt”, it also checks if all dependencies of the selected package are already present, and installs any that are missing.

If you look up on-line how to install any software in Linux, it may use “apt-get”, that is an earlier package manager, in general you can use “apt” instead now. 

The full differences between “apt” and “apt-get” depends on your Linux flavour, so this technical aside now splits further discussion by Linux flavour.

===Debian as used by Raspberry Pi===

“Debian Linux” (and its derivatives such as “Raspberry Pi Operating System”) uses “apt” to mean a ‘’’Package Manager’’’ that can install, update, and remove packages from these computer systems. 

For Debian Linux, “apt” is directed at the end-user (it has user friendly features like a staus bar showing progress on a long install or long upgrade, and can produce prompts about what it is doing and can give choices about whether to do individual actions). 

There is an alternative “apt-get” which is more powerful, but directed at system level users (those who don’t want to be watching progress and possibly responding to prompts). 

As “apt-get” is updated less frequently than “apt”, it may be it will not work with new packages. Put another way “apt-get” may never change what it can do, but “apt” may get modified to do more than it did before.

===Ubuntu===

For Ubuntu only “apt-get” was available up to 2014, when “apt” was added. Both work as described above for Debian. Again “apt” is best to use.

===Mint===

Linux Mint has a different variation. Its “apt” calls its “apt-get” and adds extra features.  So both effectively do the same, but (as with previous flavours) “apt” is best to use.

===Other Linux===

It is likely that other Linux variants will also vary how these alternative commands differ.


==Interactive Package management on RPi==

If you are using a Raspberry Pi computer, and have the full version of the Raspberry Pi Operating System that includes the desktop interactive user interface, click the "raspberry" button on your official keyboard, or the "raspberry start-here" icon in the task bar (by default task bar appears at top of screen, but can be at side or bottom; the position where the menu icon appears is also configurable) giving you access to the '''menu'''. Scroll down to preferences, then find "Add / Remove Software" item and click that.  On the top left of the screen that is displayed there is a menu item called "Options", that gives you access to the options relating to installing packages:
* '''Refresh Package Lists''' - although this is not first in list, it is the menu item that you should choose first.  You see an indication of progress, but you don't get any feedback on what it finds, which is up-to-date information about status of all your existing packages.
*  '''Check for updates''' - this menu item will pop up a "package updater" screen. If you have first selected the "Refresh Package Lists" menu item, then "Check for updates" will show any updates available for your existing packages, and you can tick/untick any in the list, and then click '''Quit''' or ''Install update'' buttons at bottom right of screen as appropriate.  This is by far the easiest way to do updates, and avoid any update where the new version might give you problems.  If you choose to update, you will be asked to supply the password.
* '''Package Log ...''' - this menu item displays (latest first) every package install or update whether done using the option above or the terminal commands described in subsequent sections.
* '''Package Sources ...''' - avoid this menu item, it should be considered an information only screen, although you could tick/untick entries there is not enough information to understand what each line represents!
The rest of the "Add / Remove Software" screen lists a lot of categories, under a search box, selecting any category causes all software in that category to be listed, component by component. None of this is friendly to novices, and I assume any technical reader can work out how to tick/untick required components. As a quick example, category "Programming" would bring up a huge list, scrolling down that list to find components starting with "php" would list what PHP components, ticking those required and the "Apply" button would install those components.

==Using Package Manager in terminal mode==

Read on for a short technical digression to explain the commands to use in a terminal session, to install and update software packages (the same commands can be used to install software for [[Your_Own_Server]], or to install software that can read the SQLite databases used by MX [[Cumulusmx.db]] and [[Diary.db]]).

==The various components to commands for installation==

Linux computers have a “source list” which references the repositories from which software packages can be installed.

If a particular package can not be found in repositories already in the source list, then another repository can be added to the source list.

===sudo===

By default, a Linux user should log in as a default user with limited rights. 

For example, the Raspberry Pi (Stretch 9, or Buster 10) Operating System has a single default user "pi", with their home folder that can be referenced as "~". (To complicate the matter, RPi Bullseye 11 operating system does not predetermine the default user name, but it does enforce setting up a default user).

The limited rights mean that a standard user cannot even see (read) some files, cannot execute some commands, and cannot edit (write to) some files.  For full access to files, the <code>sudo</code> instruction gives you (just for the command that follows) the same read/write/execute access that the system root user has.

For software packages, the command to be used is <code>apt</code> to invoke the package manager, followed by an "action" and then optionally by the "package name".  In that context the default user's limited rights allow use of actions that just read (such as '''search''', ''show''). However, for any action involving writing (such as '''install''', ''full-upgrade'', '''update''', ''autoremove''), the package manager needs additional rights, and we prefix the "apt" with '''sudo'''.

: Elsewhere on this page you might notice [[#Running any MX executable with a terminal session left open|'''cd''']] can move round the file structure (without a "sudo"), but "sudo" is used with [[# Installing/Configuring the MX service and the -service parameter|'''cp''']] as that writes a file.

Novice readers should exercise caution before using "sudo", changing folder/file ownership, adding write rights, and even using "-R" or "-y" flag. The use of "sudo" makes it very easy to inadvertently do the wrong action, in the worst case making your computer unusable because it can delete vital folders/files because of something as simple as a typo.   

: '''For technical readers only''', the full use of "sudo" is explained later (see [[#su_and_sudo]]).


===Action e.g. install===

The third part of our installation instruction is the action parameter. 

In our current context this is “install”, which tells our package manager what we are trying to either update an existing package or install a package that we did not have before,  i.e. <code>sudo apt'' install'' package_name</code>.

Here is full list of what can follow “apt” (if you are using a different package manager, I cannot guarantee that these can follow that package manager name), as we will use some of the alternatives later: 

{| class="wikitable" border="1"
|-
!style="width:30px" | Instruction following “apt”
!style="width:600px" | Description
|-
! scope="row"|update
| The "source list" [[#The various components to commands for installation|mentioned earlier]] that references the repositories from which software packages can be installed, needs to be updated periodically so it reflects any changes within those repositories. The instruction "update" is included after "apt" to make sure your computer has up to date information about repositories installed, and to report if these contain packages that can be upgraded.

In [[#Interactive Package management on RPi]] there was mention of the need to select "Update" item in menu first, and again in terminal mode, it is necessary to do this action before you do either of the next two actions.
|-
! scope="row"|install
|To upgrade or install a particular package, and its dependencies. It is mandatory to include further parts of the command after "install", see below this table.

If a later part of the command includes a package name, then this action applies to the mandatory dependencies, it does not mean all components are installed; but if a later part of the command names multiple components, then dependencies of all those components will be actioned.
|-
! scope="row"|upgrade
| Once your "source list" is up to date, <code>sudo apt upgrade</code>, will actually initiate the download of newer versions now available in repositories. 

* Note that it is not mandatory to follow this action with further parameters, but you can:
** If you include <code>-y</code> flag, or you answer 'Y' or 'y' to the prompt, then the package manager will continue, and actually replace those packages that are already installed on your computer with the downloaded newer versions, making them available for immediate use.  
** If you don't specify a package name, the default "upgrade" action will exclude anything that has had a component affected by a manual "remove" action (see below)
** It is also possible to follow the "upgrade" action (and any flag) with one or more package name(s), then the upgrade will only be done for the named packages.  (Does anyone know how to specify with flag, to exclude particular named packages?)

(You cannot upgrade the actual kernel within the operating system with this instruction, so there is no necessity to reboot your Linux computer).
|-
! scope="row"|full-upgrade
| Once your "source list" is up to date, the action "full-upgrade" can be included after "apt". The advantage of "full-upgrade" over the simple "upgrade" is that it picks up dependencies, it ensures that every package is able to work with all other packages, and so normally results in more components being upgraded. (Again, this does not affect kernel, and does not require a computer reboot).
|-
! scope="row"|autoremove
| The action "autoremove" can be included after "apt" to check all components in the packages you have installed onto your computer, and remove any components that are not needed by the dependencies of the packages you are using. A download for software frequently includes some components specifically for their software to work with particular other optional packages, and therefore installs more than you actually need.

When we install mono-complete later, the other packages we install do not need every component that mono-complete has installed, and “autoremove” can be used to tidy up when all our installations are finished.
|-
! scope="row"|remove
|If you want to remove just one component manually, use <code>sudo apt remove</code> followed by the name of component you no longer want.

One advantage of using the "remove" action is that it does not delete any related configuration files, so if we do later decide to install the package again, we don't have to repeat all configuration tasks! As a quick example, PHP is upgraded from time to time, but as an upgrade removes some system functions and adds some new ones, it is possible our existing PHP scripts will stop working if we upgrade PHP. By using the "remove" action, a future "upgrade" action will automatically exclude "php" from its default upgrades.
|-
! scope="row"|purge
| To remove any installed package and delete all related configuration files, use <code>sudo apt purge</code> then the name of the package we no longer want
|-
! scope="row"|search
|To search in repositories in source list for a package you specify after <code>apt search</code>
|-
! scope="row"|show
|To show any information available about a package that you name after the <code>apt show</code>
|}

=== Flags ===

The basic syntax is either one or two hyphens, followed by one or two letters (each letter has to be a specific case). Various examples will be seen on this page, but here just one is explained here.

If we selected "install" or "upgrade" to follow "apt", we can add a “-y” flag to signify that we want the install to not only down load new components but also install them.  Without this flag, the package manager will ask periodically if we want it to continue,  and we have to then respond with a “y” each time.  For example, when we ask to install a package, "apt" will do a search, it will list what components it has found, and output how big their demands are on storage, without "-y" flag, it will then ask (at least once) if it is okay to continue to installing.

=== Name ===

The final part of the package command is the name of the package or component that we want to install/upgrade/purge/show/remove or search for.
== A very quick introduction to Linux ==

This article is not the place to teach you Linux, you can find books and on-line articles for yourself, but I list here enough for you to understand the instructions used elsewhere in this article.

If you have a Raspberry Pi with a monitor attached, you will see a raspberry icon that you can click to get Graphical User Interface access to many features, including shutdown options.

=== su and sudo ===

There is a command <code>su</code> that allows a terminal session to become a super user session with root privileges. If you use that command, without a '''sudo''' command in front, you need to type in the password (we changed earlier) when prompted.  if you type <code>sudo su</code>, then you get root privileges without being asked to quote password. All subsequent lines in this terminal session will have a prompt that reminds you that you have root access and do not need to prefix subsequent commands with "sudo".

Normally, all terminal sessions will use the default "pi" user, and for individual commands, you will use a "sudo" prefix each time that command needs administrative rights, as this allows a standard Pi user to do tasks that otherwise only work for the root user.

You might use a "sudo" prefix if you need to access a part of the file structure that your user does not have any access to, or where the standard user does not have write (or execute) access. 

There are also some commands (like displaying mounted storage) that are not available to a standard user. Here are 3 system commands that in terminal mode will only ever work with this prefix (although if you have installed the version of the Raspberry Pi Operating System that supports a graphical user interface you can also select these actions from a menu):
*'''sudo halt''' = stops any cpu functions, but leaves Pi running; used when you have reached the end of commands you want to do for now
*'''sudo poweroff''' = makes pi do a tidy shutdown and turn off its power; used when you will not be using your Pi for a while
*'''sudo reboot''' (or "sudo reboot -verbose" for diagnostic output during shutdown and reboot) = makes your Pi close down and then reboot; used when you change settings, and after you install new software, to ensure Pi starts with all applications running using the latest settings and latest already installed software

=== ~ and / ===

The tilde symbol '''~''' denotes the home directory for the current user.  Sub-directories within the current user's folder can be identified by '''~/documents''' or similar notation. 

To reference a folder in root or any other area, the prefix is always '''/'''.

If you are using the RPi OS GUI, it provides a file manager that displays folders and files, and if you have a mouse you can click on an object to see what actions are available.  The file manager has "Home" and "Root" as bookmarks by default, you can bookmark others.  Typically, any new partitions created can also be accessed from bookmarks.  Depending on options you select, there may be icons on the GUI desktop to link to particular folders and clicking on these offers various options including opening them in file manager.

In a terminal environment, to see what files and folders are in the current directory, type <code>dir</code> for just names or <code>ls</code> for details.



=== folder commands ===

To make a new folder in the current directory, type <code>sudo mkdir folder_name</code>.

To remove a folder, that has no files in it, type in a particular path, type <code>sudo rmdir /path/directory</code>.

To remove a folder that does have files, and/or sub-folders within it, type tt>sudo rm -r /path/directory</code>, but remember the contents are gone for ever, so be absolutely sure you have specified the right folder! 

To copy folders/files from one directory to another use something like <code>cp -R --update --preserve /home/pi/CumulusMX/backup/daily /media/pi/data/CumulusMX/archive</code>

Sometimes, you have a folder or file in just one place in the file system, but want to be able to access it at a different place (because something expects it in the second place), the syntax is <code>ln -s /path/elsewhere path/pointer_location</code>.  

An example might be '''ln -s /var/lib/phpliteadmin/diary.db ~/CumulusMX/data/diary.db''' (phplite admin can only update databases in one folder /var/lib/phpliteadmin, or in older releases in /usr/share/phpliteadmin; while MX expects the file to be in its data folder but is happy with a logical pointer to another folder).

=== chmod ===

When you are attempting any of the actions listed in this article that involve reading, creating, editing, exeduting, or moving, files; you might see an error message generally because of a lack of write (or execute) permissions on an existing file or folder.   Whilst <code>rm filename</code> will remove a file even if it is write protected, for nano you need to change the file permissions with <code>sudo chmod -R ugo+rw ~/CumulusMX</code> for modify access to all files in your Cumulus installation (see the syntax below if you want to restrict access).


*'''chmod''' command to modify permissions
* the '''-R''' indicates recursive action (i.e. including not just the named folder, but all files within it and all sub-folders, and all files within sub-folders)
*letters indicating whose permission is being modified
** '''u''' = Owning user (sometimes the owner is the user root, sometimes the owner is the user Pi, for our web pages later we change ownership)
** '''g''' = Group (by default the owning user is also a group, but a group can be defined if you want to give multiple users (with different passwords) the same rights of access)
** '''o''' = Other users (write permission here is needed if for example you are using FTP to move a file from a PC to your Pi, or vice versa)
* sign for add or remove permissions
* '''+''' = add permission
* '''-''' = remove permission
*letters indicating what permission is being changed
** '''r''' = read [4]
** '''w''' = write [2]
** '''x''' = execute [1]

Note that as an alternative shorter syntax you can use numbers e.g. '''666''' is equivalent to '''ugo+rw'''. The first digit in the number relates to ''u'', the second to ''g'' and the last to ''o''. The values in [] brackets in list of permissions above are added to derive each digit.  So if you are reading the Cumulus support forum and you see a reference to permissions which includes a string of 3 digits, now you can understand what is meant.

=== editing files ===

*Do remember that file names are case sensitive.
*If you use the wrong case in a path/file name, it will be treated as a different "new" file.
*If a file editor does not display content you were expecting, look in case "new file" message appears because you have made a typo in the path/file name.

There are various text editors available on a Pi, 
*if you have a mouse and click on a file, you should see "text editor" listed, that loads '''Mousepad''' which has a menubar at the top of its "Windows" like interface.
*in terminal mode '''nano''' is a text editor that by default lists the actions available making it easier for a novice to use. 
*in both the GUI and terminal mode, Geary is a programmer's editor with lots of useful funtionality

All editors can create a file when a file does not exist and edit (subject to file permissions) an existing file. Use prefix of 'sudo' to give you access to any file irrespective of ownership, '''sudo''' does not change the actual file permissions, so you might find you can read a file, but not save it after you have done your edit.  

====nano====

The full syntax is <code>sudo nano -B Path_file_name</code> where the '''-B''' means it will create a backup of how the file was before (this can be enabled while in the editor by pressing the control key down and typing B). Alternatively use '''-C''' which stores each version in a back-up directory.  If you want to edit from a particular line and column you can use '''+line.column''', and also optionally use '''-l''' (lower-case "L") to display line numbers which might be useful when trying to correct a problem with a log file like [[dayfile.txt]].  If you don't specify a file name, then nano will create a new file and you will need to specify where to save it before exit.

After typing the nano command you need to specify a filename (it might include a path, see earlier sub-section for use of '''/''' and '''~''') and there are examples later in this article, but if you decide to ''host a web site on your Pi'' then you might want to edit its home page with (.html or .php) name like <code>sudo nano /var/www/html/index.php</code>.  

After you have made an alteration to the current contents of the file, various options are shown at the bottom. Here are two key ones:
*First is '''^O''' which is used to save the file whilst staying in the editor, to do this press the control key down and type O. Next it shows the current file name, if you press '''Enter''' then that file will be overwritten.
**it allows you to type over the file name shown. If you choose to save as another file, you will be asked if the new name is correct (type '''Y''' to continue saving).
*Another is '''^X''' which means if you press the control key down and type X you get the exit dialogue. If you have not made any edits, or have already saved the file, this just exits the editor. If you have not used control and O to save the file, it asks whether you want to save the edited file (type '''Y'''), typing just the Y key lets save continue (any other key stroke exits without saving), then it shows the current file name, if you press '''Enter''' then that file will be overwritten.

You might find it useful to type <code>sudo nano /etc/nanorc</code> as this puts you into the configuration file for nano where you can set back-up, line-numbering, and other options.

====Geany====

This uses a GUI, you can set preferences and do all other actions using either menu selections (use mouse or keyboard) or control sequences (on keyboard).  Once it knows what type of programming language, it can colour up the code; it can show you how many times variable identifiers are used; it can match opening and closing quotes, tags, and brackets; and it can ensure encoding and line terminators are correct.

Line-numbering is an option, so it can be used to edit MX log files, and (as BOM is an encoding option) you can be sure it won't add unwanted encoding.

== removing an unwanted file ==

You can remove a file with various commands, including <code>sudo rm filename</code>.