Cumulus Wiki

Charts Definition Language (CDL)

Revision as of 09:45, 16 September 2021 by HansR (talk | contribs) (→‎Example CutilsCharts.def)

Introduction

The Charts Definition Language gives the user the possibility to define his own charts on the basis of the CumulusMX infrastructure through the ChartsCompiler. This article gives the specification of that language.

CutilsCharts.def

The charts are defined in the definition file (CutilsCharts.def which resides in the utils directory) which contains one or more chart specifications according to the figure below. All attributes of a plot variable are optional and if not specified have a reasonable default. All white space characters are used as separators so it is easy to make a visually attractive, readable definition file.

The text in the CutilsCharts.def is case independent, however for clarity CamelCase or a variety thereof is advised.

The ChartsCompiler is obligatory after the introduction of the ExtraSensors module from version 6.5 onwards where the ChartsCompiler is used internally for defining and creating the charts for the ExtraSensors module which would otherwise - in a generic way - be virtually impossible. The CutilsCharts.def has a demarcation line which separates the Extra Sensors Charts (below) from the user charts (above). The user can define he wants to create his own Extra Sensor Charts by setting the parameter UserModificationExtraSensorCharts=true.

A standard CutilsCharts.def and a CutilsCharts-example.def are in the distribution. The first for an initial setup, the second to guide the user into the possibilities.

The specification

Chart Compiler Language (CDL) specification


The Language

The Chart block

 CHART <id> TITLE [<word>...<word>]  CONNECTSto [1 … 24]

 [ PLOT [ALL||DAILY|RECENT|EXTRA] <Plotvariable> [ EVAL '[' <Expression> ']' ]
            AS <Line Type> 
            COLOUR <HTML Color>
            ZINDEX <Integer>
            LINEWIDTH <Integer>
            OPACITY <Integer>.<Integer>
            AXIS <Axis Type>
 ]
 [ STATS <Plotvariable> <Stats Type> ]

 PLOT blocks repeated
 STATS blocks repeated
   [...]

 ENDCHART OUTPUT <filespec>

 CHART […] ENDCHART blocks repeated

Plot blocks can be repeated as many times as you wish but different parameters can result in many yAxis (making the plot area smaller) and the chart may become crowded and unreadable. Experiment and try-out.

Syntactically the STATS block can be many as well but it requires semantically the Plotvariable of which you want statistics to be plotted normally. So you can have e.g. three STATS blocks for one PLOT block.

Output specifier

An output file can be specified for each chart by putting OUTPUT <filename> after an ENDCHART. All charts from that point on will be written to that specified output until a new output is specified for another chart. If no output is specified at all, all charts will be written to cumuluscharts.txt (the default output)

If you are using the website it is important to know that the first chart will always be used as the chart for the Home page button and that cannot be changed (the file is always cumuluscharts.txt). If, however you specify an output on the first chart, that chart will be duplicated in the output you specify.

The output specifier gives the user the possibility to define charts as he likes and to place these in the menu wherever he likes. In the Website Generator the placements of the charts at the moment will be limited to the User Defined Menu. In modular use the user can do whatever he likes with the resulting files. In the website the charts require the datafiles to be in the current directory.

Keywords

All capital written words in the Chart block section are keywords (reserved words) and have meaning.

Explanation of the definition Keywords

  1. EQUATIONS: Starts an equations block. Such block only defines (partial) equations, it does not cause code generation in itself
  2. CHART: Starts a chart definition
  3. TITLE: Starts the title. All words until the next keyword comprise the title of the chart
  4. CONNECTSTO: The integer numbers which follow CONNECTSTO represent a pane in the realtime dashboard. When clicked on that pane this chart appears. This is only valid on charts in the default output. For other outputs there is no semantics for CONNECTSTO and a warning is issued in the logfile if it is specified in any second or higher output.
  5. PLOT: Starts the plot specification using one of the possible variables
  6. STATS: Starts the statistics block using one of the possible variables. Semantics demands the variable to be plotted normally as well.
  7. ALL: Defines the variable to be taken from the Dayfile and all data for that variable will be plotted (Note that the CMX parameter ChartMaxDays will not be used)
  8. RECENT: Defines the variable to be governed by the CMX parameter GraphHours and are similar to the CMX RECENT graphs (though not equal).
  9. AS: Defines the type of charting for that variable.
  10. EVAL [ ]: Defines an expression for the plot variable. The equation must be placed between the square brackets, space has no value here (does not act as separator).
  11. COLOUR: Defines the colour of that variable.
  12. ZINDEX: Defines the zIndex plane for that variable.
  13. LINEWIDTH: Defines the linewidth for that variable.
  14. OPACITY: Defines the opacity for that variable (important for area and column graphs)
  15. AXIS: Gives the user the possibility to specify an axis when using the EVAL construct otherwise an error is generated.
  16. ENDCHART: End the chart definition
  17. OUTPUT: Gives an optional name of an outputfile which starts at the chart where it is defined. Any chart after this OUTPUT specifier goes to this OUTPUT until another OUTPUT is found. Only the first OUTPUT specifier accepts click events as defined by CONNECTSTO.

The keywords for the actual parameters and the Graph Types speak for themselves.

Plotvariables

The following PlotVariables are reserved words:

With the RECENT qualifier
     "InsideTemp", "Dewpoint", "ApparentTemp", "FeelsLike", "WindChill", "HeatIndex","Temperature", "Humidex",
     "WindGust", "WindSpeed",
     "Bearing", "AverageBearing",
     "UV", "SolarRadiation", "TheoreticalSolarMax",
     "RainFall", "RainRate",
     "Pressure",
     "Humidity", "InsideHumidity",
     "EvapoTranspiration"
With the ALL/DAILY qualifier
     "MinTemp", "MaxTemp", "AverageTemp", "WindChill", "MaxDewpoint", "MinDewpoint","MaxFeelsLike", "MinFeelsLike", 
     "MaxGust", "WindRun", "WindSpeed", 
     "UV-Index", "SolarRadiation", "SunHours",
     "RainFall", "MaxRainRate",
     "MinBarometer", "MaxBarometer",
     "MinHumidity", "MaxHumidity",
     "HeatingDegreeDays","CoolingDegreeDays","EvapoTranspiration"

Note: The difference between ALL and DAILY is only between the displayed datarange (6 months or 1 month)

With the EXTRA qualifier
     "Temp1","Temp2","Temp3","Temp4","Temp5","Temp6","Temp7","Temp8","Temp9","Temp10",
     "Humidity1","Humidity2","Humidity3","Humidity4","Humidity5","Humidity6","Humidity7","Humidity8","Humidity9","Humidity10",
     "Dewpoint1","Dewpoint2","Dewpoint3","Dewpoint4","Dewpoint5","Dewpoint6","Dewpoint7","Dewpoint8","Dewpoint9","Dewpoint10",
     "SoilTemp1","SoilTemp2","SoilTemp3","SoilTemp4","SoilTemp5","SoilTemp6","SoilTemp7","SoilTemp8","SoilTemp9","SoilTemp10",
         "SoilTemp11","SoilTemp12","SoilTemp13","SoilTemp14","SoilTemp15","SoilTemp16",
     "SoilMoisture1","SoilMoisture2","SoilMoisture3","SoilMoisture4","SoilMoisture5","SoilMoisture6","SoilMoisture7","SoilMoisture8","SoilMoisture9","SoilMoisture10",
         "SoilMoisture11","SoilMoisture12","SoilMoisture13","SoilMoisture14","SoilMoisture15","SoilMoisture16",
     "AirQuality1","AirQuality2","AirQuality3","AirQuality4",
     "AirQualityAvg1","AirQualityAvg2","AirQualityAvg3","AirQualityAvg4",
     "UserTemp1","UserTemp2","UserTemp3","UserTemp4","UserTemp5","UserTemp6","UserTemp7","UserTemp8",
     "LeafTemp1","LeafTemp2","LeafWetness1","LeafWetness2",
     "CO2", "CO2_24h", "CO2_pm2p5", "CO2_pm2p5_24h","CO2_pm10","CO2_pm10_24h","CO2_temp","CO2_hum"

Note: The EXTRA qualifier indicates plotvariables for the Extra Sensors. These are implicit RECENT but come from a different source.

Types

Graph Types

The following graphs are possible 

     Line, SpLine, Area, Scatter, Column, ColumnRange

All graph types can be combined in a chart except for Scatter.

ColumnRange only works for DAILY and ALL range specifiers. If the Max and Min lines of the range must be shown, they need to be specified as a separate line. The range can be specified for either Min or Max, the range will automatically be applied and no line will be shown. The ColumnRange is valid for all variables in the datafile with a Max/Min range: Temperature, Dewpoint, FeelsLike (and other derivatives), Barometer and Humidity. Other variables will generate an error.

Axis Types

The following Axis are possible. 

     Temp, Wind, Distance, Hours, Solar, UV, Rain, Rrate, Pressure, Humidity, DegreeDays, EVT, AQ, ppm

This only has meaning if there is an EVAL with a valid expression present. As an EVAL may modify the value of the plot variable to something completely different, the original Axis belonging to that plot variable may be useless or scale wrongly. Therefore you can choose your own Axis. If there is no valid equation, a warning is given and no axis is generated.

In future a generic (without title or unit) Axis will selectable to accommodate plotting in combination with other variables.

Other Types

For other types such as STATS types (or methods) and FUNCTION types for EVAL, see those respective pages for STATS and for EVAL.

Equations

See page CDL - EVAL.

The Chart Axis

All PlotVariables have their own defined axis and those axis will be coupled automatically to the chart. Multiple axis will be alternately added left-right where the order in the definition files determines this order. So it is the user who ultimately can define where the axis are places: left or right and from inside to the outside in the order where they are encountered during the parsing process.

The following axis are currently defined and implicit for the plot variables (the user has no options here):

  1. None
  2. Temp
  3. Pressure
  4. Rain
  5. Rrate
  6. Wind
  7. Direction
  8. Humidity
  9. Solar
  10. UV
  11. Hours
  12. Distance
  13. DegreeDays
  14. EVT
  15. Free (has no title, unit or context associated with it. It just displays values)
  16. AQ
  17. ppm

Translations for the axis titles come from the [Website] section and should require no additional effort.

Additional details

  1. All parameters can be combined in one graph with the corresponding axis automatically added. Only when scatter is defined as Line Type it is forbidden to combine it with one of the other possibilities. This may change in future.
  2. The Legend texts are implemented as translations of the PlotKeywords (the Plot Variables). You will find these in the translation file section [Compiler]. Legend entries can be multiple words (the keywords can not).
  3. The width of the graph container is 100% of the viewport.
  4. The height of the graph container can be specified in the options (not defined in the language to prevent overcharge):
 [Compiler]
 ChartContainerHeight=650 (Default)
  1. The Plot Variables Evapotranspiration, CoolingDegreeDays and Heating DegreeDays are not supplied by CMX in the JSONs. CumulusUtils provides those automatically when used in the definitions (on request) with the commandline parameter UserAskedData. See also ChartsCompiler Operation.
  2. You have to schedule this command if you wish to use the FTP frequency CMX uses for its JSON's. Note that these JSON files specific to CumulusUtils always reside in the website root of CumulusUtils, even if you have more websites and define the CumulusRealTimeLocation.
  3. If the user uses different locales for CMX and CumulusUtils the monthly log filenames will be according to the locale of CMX. This creates an issue which is solved by the inifile parameter:
 [General]
 MonthsOfMiracleAndWonder=jan,feb,mrt,apr,mei,jun,jul,aug,sep,okt,nov,dec

Default this variable contains the abbreviated month names according to the locale of CumulusUtils. The user may edit those names to the ones used by CMX.

Example CutilsCharts.def

The Chart definitions below is included in the distribution of CumulusUtils and serves as an example.

ChartsCompiler Demo and test chart definitions - Part of CumulusUtils
© Copyright 2019 - 2021 Hans Rottier <hans.rottier@gmail.com>
Author
Hans Rottier <hans.rottier@gmail.com>
Project
CumulusUtils meteo-wagenborgen.nl
Date
04-08-2021

Equations

Water Vapour Saturation Pressure according to the August-Roche-Magnus equation for the 1980 version, the 2008 version and the sub-zero version
 Psat2008 EVAL [ 6.112 * EXP(17.62*Temperature/(243.12+Temperature)) ]
 Psat2008ALL EVAL [ 6.112 * EXP(17.62*AverageTemp/(243.12+AverageTemp)) ]
 Psat1980 EVAL [ 6.1094 * EXP(17.625*Temperature/(243.04+Temperature)) ]
 PsatSub0 EVAL [ 6.112 * EXP(22.46*Temperature/(272.62+Temperature)) ]
Actual Vapour Pressure
 Avp Eval [ Humidity / 100 * Psat2008 ]
Humidex as in CMX
 CDLHumidex Eval [ (Temperature + 5.0 / 9.0 * (Psat2008 - 10.0)) ]
Do the WindChill According to https://sciencing.com/relative-humidity-7611453.html / CMX
Note the eqn is only valid for degr Celsius and Kmh.
See https://sciencing.com/relative-humidity-7611453.html for imperial formula.
WindPOW Eval [ pow(WindSpeed, 0.16) ]
CDLWindChill Eval [ 13.12 + 0.6215 * Temperature - 11.37 * WindPOW + 0.3965 * Temperature * WindPOW ]
Apparent Temperature as in CMX (WindSpeed in ms so convert to Kmh, divide by 3.6
 AppTemp Eval [ Temperature + 0.33 * Avp - 0.7 * WindSpeed / 3.6 - 4.0 ]
WetbulbTemperature in degr Celsius
 WetBulbTemp Eval [ 0.00066 * Pressure * Temperature + 4098 * Psat2008 / (Pow( Dewpoint + 237.7, 2) * Dewpoint) / (0.00066 * Pressure + 4098 * Psat2008 / Pow(Dewpoint + 237.7, 2)) ]


Chart Overzicht Title Overzicht van de huidige weerstatus ConnectsTo 2 4 15 20 

 PLOT Temperature Colour #058DC7 zindex 90
 PLOT DewPoint Colour #50B432 zindex 80
 PLOT pressure Colour red zindex 70
 Plot Humidity Colour #64E572 zindex 60
 Plot SolarRadiation As Area Opacity 0.2 Colour Orange zindex 20
 Plot RainFall As Area colour #33B4FF opacity 0.2 zindex 10

EndChart

Chart Temperatuur Title Temperatuur en afgeleiden ConnectsTo 1 13 14 

 Plot InsideTemp As Spline Colour #DDDF00 zIndex 100
 PLOT Temperature As SPline Colour #058DC7 zindex 90
 PLOT FeelsLike As SPline Colour #ED561B zindex 80
 PLOT DewPoint As SPline Colour #50B432 zindex 70

EndCHART

Chart Wind Title Wind ConnectsTo 5 8 

 Plot WindGust
 Plot WindSpeed

EndCHART

Chart Windrichting Title Windrichting ConnectsTo 6 7 

 Plot Bearing As Scatter
 Plot AverageBearing As Scatter

EndCHART

Chart Regen Title Regen and Regensnelheid ConnectsTo 3 21 22 

 Plot RainFall As Area Colour green Opacity 0.2
 Plot RainRate zIndex 100

EndChart

Chart Vochtigheid Title Relatieve vochtigheid binnen en buiten ConnectsTo 11 16 

 Plot Humidity
 Plot InsideHumidity

EndChart

Chart Zon Title (Max)Straling en UV ConnectsTo 12 23 24 

 Plot	UV	zIndex 100
 Plot 	SolarRadiation As Area Opacity 0.4 Colour Orange zIndex 50 
 Plot	TheoreticalSolarMax As Area Colour Gold zIndex 10 Opacity 0.2

EndChart

CHART DagelijkseTemp TITLE Dagelijkse gem. temperatuur met columnrange 

 PLOT DAILY AverageTemp colour green As ColumnRange
 PLOT DAILY AverageTemp colour black As spline

EndChart

CHART DagelijkseBaro TITLE Dagelijkse gem. barometer met columnrange 

 PLOT DAILY MinBarometer colour Crimson As ColumnRange
 PLOT DAILY AverageBarometer EVAL [ (MinBarometer + MaxBarometer)/2 ] colour black As spline Axis Pressure

EndChart

CHART DagTemp TITLE Dagelijkse Gemiddelde Temperatuur met Min/Max van afgelopen jaar 

 PLOT DAILY AverageTemp colour green
 Plot DAILY MinTemp colour blue
 Plot DAILY MaxTemp colour red

EndChart OUTPUT AllChart.txt

Chart DagRegen Title Dagelijkse regenval van afgelopen jaar 

 Plot DAILY RainFall As Column colour lightblue

EndChart

Chart DagZonuren Title Zonuren per dag van afgelopen jaar 

 Plot ALL SunHours As Column colour gold

EndChart

Chart TempEVT Title Temperatuur en EVT - Historische Data ConnectsTo 12 23 24 

 PLOT ALL AverageTemp As SPline Colour #058DC7 zindex 90 LineWidth 10
 PLOT ALL Evapotranspiration As Column Colour #cc0000

EndCHART OUTPUT RarityCharts.txt

Chart TempEVTRecent Title Temperatuur en EVT - Recente Data 

 PLOT Temperature As SPline Colour #058DC7 zindex 90 LineWidth 10
 PLOT Evapotranspiration As Area Colour #cc0000

EndCHART

Chart CoolingDays Title Overige variabelen 

 Plot All AverageTemp
 Plot ALL CoolingDegreeDays
 Plot ALL HeatingDegreeDays
 Plot All Evapotranspiration

EndChart

CHART DailyTemp TITLE Dagelijkse temp met rangecolumn 

 PLOT DAILY AverageTemp colour green As ColumnRange
 PLOT DAILY AverageTemp colour black As spline

EndChart OUTPUT ColumnRange.txt

CHART DailyBarometer TITLE Dagelijkse Barometer met rangecolumn 

 PLOT DAILY MinBarometer colour Crimson As ColumnRange
 PLOT DAILY AverageBarometer EVAL [ (MinBarometer + MaxBarometer)/2 ] colour black As spline Axis Pressure

EndChart

CHART DailyTempStats TITLE Temp with statistics offset 5 

 PLOT ALL AverageTemp colour pink
 PLOT ALL sumAverageTemp EVAL [ Sum(AverageTemp-5) ] COLOUR green AXIS DegreeDays
 STATS ALL AverageTemp SMA COLOUR crimson

EndChart Output Statistics.txt

CHART DailyRainStats TITLE Rain with statistics 

 PLOT ALL RainFall colour CornflowerBlue as column 
 PLOT ALL sumRainFall EVAL [ SUM( RainFall) ] COLOUR crimson AXIS Free
 STATS ALL RainFall SMA colour green

EndChart

Documentation
https://en.wikipedia.org/wiki/Dew_point
https://iridl.ldeo.columbia.edu/dochelp/QA/Basic/dewpoint.html
Compare the Dewpoint from Cumulus with the calculated approximation
Later I will enter the Cumulus Calculation and have the Humidity be produced by the Davis station
Nice comparison

Chart Dewpoint Title Dewpoint Calculations in CDL 

 Plot Humidity
 Plot Dewpoint
 Plot Approximation1 EVAL [ (243.12 * LN(Psat2008) - 440.1) / (19.43 - LN(Psat2008)) ] Axis Temp
 Plot Approximation2 EVAL [ Temperature - ((100 - Humidity)/5) ] Axis Temp
STATS Dewpoint SMA colour green

EndChart

Chart VapourPressure Title August-Roche-Magnus Eqn 1980/2008/SubZero in CDL 

 Plot Temperature
 Plot Psat2008 Eval [ Psat2008 ] Axis Free
 Plot Psat1980 Eval [ Psat1980 ] Axis Free
 Plot PsatSub0 Eval [ PsatSub0 ] Axis Free
 Plot Avp Eval [ Avp ] Axis Free

EndChart

Chart TempDerivatives Title Temperature Derivatives full in CDL 

 Plot Temperature
Plot CDLWindChill Eval [ CDLWindChill ] Axis Temp
 Plot AppTemp Eval [ AppTemp ] Axis Temp
 Plot WetBulbTemp Eval [ WetBulbTemp ] Axis Temp
 Plot CDLHumidex Eval [ CDLHumidex ] Axis Temp

EndChart

Chart pwsFWI Title Raw pwsFWI as a chart 

 Plot DAILY pwsFWI Eval [ Psat2008ALL * (1 - MinHumidity/100)*HighAvgWindSpeed ] As spline Axis Free Colour Crimson

EndChart Output pwsFWIchart.txt

Chart pwsFWInow Title Raw pwsFWI intraday as a chart 

 Plot RECENT pwsFWI Eval [ Psat2008 * (1 - Humidity/100)*WindSpeed ] As spline Axis Free Colour Red

EndChart

Retrieved from "https://www.cumuluswiki.org/index.php?title=Charts_Definition_Language_(CDL)&oldid=9921"