---
title: "Generating STICS XML files from tabulated data"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Generating STICS XML files from tabulated data}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r eval=FALSE}
library(SticsRFiles)
library(dplyr)
```

```{r include=FALSE}
suppressWarnings(library(SticsRFiles))
```

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  eval = FALSE,
  workspace_path <- tempdir(),
  stics_version <- SticsRFiles::get_stics_versions_compat()$latest_version
)
```




# Introduction

* In this part information are focusing on functionalities related to STICS XML files generation capabilities based on parameterization for lists of simulations (USMs) stored in tabulated data files; as Excel or CSV files.

* The functions operate on data loaded as data.frames from Excel sheets or CSV files

* Specialized functions for producing XML files are taking into account: **initialization files**, **stations files**, **crop management files**, **usms file** and **soils files**. General parameters files generation are not available, but will be eventually included in future developments.

* Observations data files generation have been included even if they are not in XML format.

* The generation of weather files is not included yet in the package, but will be included in a future version. One can use the tool provided by JavaSTICS GUI to perform these files generation.


This document has been generated using the latest STICS version **`r stics_version`**.


# Package functions to be used {#pkg-funs}

There is not at the moment an integrated function which is able to produce all needed parameters files. Individual functions are dedicated to each parameter file kind and may be called individually or gathered in a single script for doing all tasks at once. Use examples are given in next parts below.

* gen_usms_xml: for generating a `usms.xml` file
* gen_sols_xml: for generating a `sols.xml` file
* gen_ini_xml: for generating simulations initialization files (`*_ini.xml`)
* gen_tec_xml: for generating crop management files (`*_tec.xml`)
* gen_sta_xml: for generating weather stations files (`*_sta.xml`)
* gen_general_param_xml: for generating general parameters files (`param_gen.xml and param_newform.xml`)





# Getting a table file example {#download-excel-example}

## Getting inputs `Excel` example file from the `SticsRFiles` package {#get-xl}

In the following example `/path/to/target/dir` must be replaced with a real path depending on the operating system (i.e. starting with `c:/` for windows for example)

```{r eval=FALSE}
# Getting the file in the current directory
usm_xl_file <- download_usm_xl()

# Getting the file in a specic directory
xl_dir <- "/path/to/xl/dir" # or something like C:/path/to/xl/dir" for Windows

usm_xl_file <- download_usm_xl(out_dir = xl_dir)

#> [1] "inputs_stics_example.xlsx  has been copied in directory
#> "/path/to/xl/dir

```

```{r, eval=TRUE, echo = FALSE, results='hide'}
usm_xl_file <- download_usm_xl(file = "inputs_stics_example.xlsx",
                               out_dir = workspace_path)
```




## Getting an USMs inputs `CSV` example file from the `SticsRFiles` package {#get-csv}

```{r eval=FALSE}
# Getting the file in the current directory
usm_csv_file <- download_usm_csv(file = "inputs_stics_example_USMs.csv")

# Getting the file in a specific directory
csv_dir <- "/path/to/csv/dir" # or something like C:/path/to/xl/dir" for Windows

usm_csv_file <- download_usm_csv(file = "inputs_stics_example_USMs.csv",
                                 out_dir = csv_dir)

#> [1] inputs_stics_example_USMs.csv  has been copied in directory
#> "/path/to/csv/dir
```

```{r, eval=TRUE, echo = FALSE, results='hide'}
usm_csv_file <- download_usm_csv(file = "inputs_stics_example_USMs.csv",
                                 out_dir = workspace_path)
```





# Description of tables expected content and use

As told in the previous section, parameters data may be stored either in Excel files containing several sheets or in several equivalent individual CSV files.

Sheets content and CSV files content is the same, only their use is different. But, CSV files are easily traced under version control systems and they also can be generated programmatically with the help of functions dedicated for example to generating crop management parameters like irrigation, fertilization (interventions and doses calculations).


## Important information

The USMs data which are necessary for generating the `usms.xml` must be all provided either through an Excel sheet or a CSV file; as shown in the [USMs sheet example](#usms-table-example).

Other data stored in Excel or CSV files must be adapted to the use cases here after:

* For generating new files, containing parameters with different values for each USM, tables **must contain values for all the files parameters**.

* For generating files with a common set of parameter values and only a part of them must be overloaded with values coming from parameters tables, one **must give an existing** file as a template for files generation to functions `gen_*_xml` (listed [here](#pkg-funs)).


## Specificities of Excel sheets

Excel sheet names are completely free, but must be meaningful regarding the kind of parameters set they contain.

In the example, useful sheets for generating XML parameters files are named: **USMs, Ini, Soils, Tec, Station**






## Common tables specifications

The **first column** is used for storing identifiers or files names (or prefix) and the USMs table contains identifiers of the other linked tables in columns names **finit, nomsol, fstation, ftec_1, ftec_2** with respect to initialization files, soil name, station files and crop management files (for main crops and optionally associated crops).

The **first column names** of data tables are automatically detected and must contain keywords prefix to do so as listed: **usm_, ini_, soil_, sta_, tec_**

These first columns may contain either **names identifiers or files names** according to data type:

* USMs or soil data: usm name or soil name
* Initialization, station or crop management data: files names either with or without suffix and XML extension
    (**\_ini.xml, \_tec.xml, or \_sta.xml**)

Other columns of tables are named with **parameters names** according to the following rules:

* Scalar parameters
used parameter names must match the exact parameter names (checking it syntax with the help of the `get_param_info` function)
* Vector parameters
used parameter names must also match real parameter names with adding a suffix indicating their index; for example jultrav_1, jultrav_2, and so on…
* Exceptions
in some cases used parameters names in tables may be different from real parameter names

Here is the actual **correspondence table** of `keywords` to be used in parameters tables (i.e excel files for example) in replacement of real names in XML files:

```{r echo=FALSE, eval=TRUE, fig.align='center'}
l <- SticsRFiles:::get_params_dict()
df <- data.frame(keyword = names(l), realname = unlist(l, use.names = FALSE),
                 stringsAsFactors = FALSE)
rmarkdown::paged_table(df)
```


As an example, looking in the Tec sheet of the previously downloaded Excel file, irrigation parameters for setting days and amount of water supplied, are respectively named julapI_1, julapI_2,… and doseI_1, doseI_2,… instead of using **absolute_value/%** and **amount**.



## Data consistency checks in tables

For the moment, *no consistency checks* (between Excel sheets or CSV equivalents files) functionalities have been included in the package in order to prevent missing generated files or misspelled file names or identifiers.

So, one must take care of **links between xml files names** for example files names defined for usms related to names declared for example in Excel sheets `Tec`, `Station`, and so on.

In future developments, checks will be included to run be run prior to files generations, for preventing errors.


## USMs table example {#usms-table-example}

The Excel file which contains parameters values to take into account for STICS inputs may contain specific sheets for each kind of parameters groups, i.e. USMs, crop management, soil, weather station, initializations (except plant parameters and general parameters).

The following table contains data extracted from a sheet named USMs containing a list of USMS parameters or from his equivalent CSV example file.


```{r, eval = TRUE, echo = FALSE}
library(readxl)
xl_param <- read_params_table(usm_xl_file, sheet = "USMs")
knitr::kable(xl_param)
```


# Generating the `usms` XML file (*usms.xml*)

The gen_usms_xml function is working on a data.frame object that can be loaded either from an Excel file sheet or from a csv file. Here after code shows how to get data from the downloaded examples [Excel](#get-xl) and [CSV](#get-csv) files.

## Getting data


<!-- Only for the show -->
```{r read_xl_file_usms, eval = FALSE}
# Reading the Excel file
usms_param <- read_params_table(usm_xl_file, sheet_name = "USMs")
# Or
# Reading the CSV file
usms_param <- read_params_table(usm_csv_file)
```


<!-- active chunk to be run -->
```{r eval_read_xl_file_usms, eval = TRUE, echo = FALSE}
library(readxl)

# Reading the Excel file
xl_param <- read_params_table(usm_xl_file, sheet_name = "USMs")

xl_param <- read_params_table(usm_xl_file, sheet_name = "USMs")

rmarkdown::paged_table(xl_param)
```

## Generating the file


<!-- Only for the show -->
```{r gen_usms_file, eval = FALSE}

# Output file path
out_file <- "/path/to/file/usms.xml"
# or something like C:/path/to/file/usms.xml" for Windows

# Generating a new usms.xml file, for all xl_param lines
gen_usms_xml(file = out_file, param_df = usms_param)
```

<!-- active chunk to be run -->
```{r eval_gen_usms_file, eval = TRUE, echo = FALSE}
out_file <- file.path(workspace_path, "usms.xml")

# Generating a new usms.xml file, for all xl_param lines
gen_usms_xml(file = out_file, param_df = xl_param)
```

## `usms.xml` file content

Example of the first `usm` parameters:

```{r show_usms_file, eval = TRUE, echo = FALSE, class.output="xml"}
l <- readLines(con = out_file)
cat(paste(c(l[1:24], "...", l[length(l) - 1]), collapse = "\n"))
```


# Generating the `sols` XML file (*sols.xml*)

As for `gen_usms_xml`, `gen_sols_xml` function is working on a data.frame object loaded from an Excel file sheet. The following code only shows here after the `sols.xml` file generation section.


## Getting data

<!-- Only for the show -->
```{r read_xl_file_sols, eval = FALSE}
# Reading the Excel file
soils_param <- read_params_table(usm_xl_file, sheet_name = "Soils")
```


<!-- active chunk to be run -->
```{r eval_read_xl_file_sols, eval = TRUE, echo = FALSE}
library(readxl)

# Reading the Excel file
soils_param <- read_params_table(usm_xl_file, sheet_name = "Soils")

rmarkdown::paged_table(soils_param)
```

## Generating the file

<!-- Only for the show -->
```{r gen_sols_file, eval = FALSE}

# Output file path
out_file <- "/path/to/file/sols.xml"
# or something like C:/path/to/file/usms.xml" for Windows

# Generating a new sols.xml file, for all xl_param lines
gen_sols_xml(file = out_file, param_df = soils_param)
```

<!-- active chunk to be run -->
```{r eval_gen_sols_file, eval = TRUE, echo = FALSE}
out_file <- file.path(workspace_path, "sols.xml")

# Generating a new sols.xml file, for all xl_param lines
gen_sols_xml(file = out_file, param_df = soils_param)
```

## `sols.xml` file content


Example of the first `sol` parameters sub-list:

```{r show_sols_file, eval = TRUE, echo = FALSE, class.output="xml"}
l <- readLines(con = out_file)
cat(paste(c(l[1:30], "...", l[110], "...", l[length(l) - 1]), collapse = "\n"))
```


# Generating multiple crop management XML files (*\*\_tec.xml*)

## Getting data
<!-- Only for the show -->
```{r read_xl_file_tec, eval = FALSE}
# Reading the Excel file
tec_param <- read_params_table(usm_xl_file, sheet_name = "Tec")
```


<!-- active chunk to be run -->
```{r eval_read_xl_file_tec, eval = TRUE, echo = FALSE}
library(readxl)

# Reading the Excel file
tec_param <- read_params_table(usm_xl_file, sheet_name = "Tec")

rmarkdown::paged_table(tec_param)
```

## Generating files

<!-- Only for the show -->
```{r gen_tec_files, eval = FALSE}
# *_tec.xml files, one for each xl_param line
gen_tec_xml(param_df = tec_param, out_dir = workspace_path)
```

<!-- active chunk to be run -->
```{r eval_gen_tec_files, eval = TRUE, echo = FALSE}

# *_tec.xml files, one for each xl_param line
gen_tec_xml(param_df = tec_param, out_dir = workspace_path)
```

Example of a `tec` parameters file content sub-list:

```{r show_tec_file, eval = TRUE, echo = FALSE, class.output="xml"}
l <- readLines(con = file.path(workspace_path,
                              "BIN_CAN_05_SEC_220-0-0_34K_CANPC05T3_Q_tec.xml"))
cat(paste(c(l[1:30], "...", l[length(l) - 1]), collapse = "\n"))
```


# Generating multiple initialization XML files (*\*\_ini.xml*)

## Getting data
<!-- Only for the show -->
```{r read_xl_file_ini, eval = FALSE}
# Reading the Excel file
ini_param <- read_params_table(usm_xl_file, sheet_name = "Ini")
```


<!-- active chunk to be run -->
```{r eval_read_xl_file_ini, eval = TRUE, echo = FALSE}
library(readxl)

# Reading the Excel file
ini_param <- read_params_table(usm_xl_file, sheet_name = "Ini")

rmarkdown::paged_table(ini_param)
```

## Generating files

<!-- Only for the show -->
```{r gen_ini_files, eval = FALSE}
# *_ini.xml files, one for each xl_param line
gen_ini_xml(param_df = ini_param, out_dir = workspace_path)
```

<!-- active chunk to be run -->
```{r eval_gen_ini_files, eval = TRUE, echo = FALSE}

# *_ini.xml files, one for each xl_param line
gen_ini_xml(param_df = ini_param, out_dir = workspace_path)
```
## XML file content

Example of a `ini` parameters file content sub-list:

```{r show_ini_file, eval = TRUE, echo = FALSE, class.output="xml"}
l <- readLines(con = file.path(workspace_path, "USM_2017_T1_ini.xml"))
cat(paste(c(l[1:19], "...", l[length(l) - 1]), collapse = "\n"))
```






# Generating multiple meteorological station XML files (*\*\_sta.xml*)

## Getting data
<!-- Only for the show -->
```{r read_xl_file_sta, eval = FALSE}
# Reading the Excel file
sta_param <- read_params_table(usm_xl_file, sheet_name = "Station")
```


<!-- active chunk to be run -->
```{r eval_read_xl_file_sta, eval = TRUE, echo = FALSE}
library(readxl)

# Reading the Excel file
sta_param <- read_params_table(usm_xl_file, sheet_name = "Station")

rmarkdown::paged_table(sta_param)
```

## Generating files

<!-- Only for the show -->
```{r gen_sta_files, eval = FALSE}
# *_sta.xml files, one for each xl_param line
gen_sta_xml(param_df = sta_param, out_dir = workspace_path)
```

<!-- active chunk to be run -->
```{r eval_gen_sta_files, eval = TRUE, echo = FALSE}

# *_sta.xml files, one for each xl_param line
gen_sta_xml(param_df = sta_param, out_dir = workspace_path)
```
## XML file content

Example of a `sta` parameters file content sub-list:

```{r show_sta_file, eval = TRUE, echo = FALSE, class.output="xml"}
l <- readLines(con = file.path(workspace_path, "climatex_sta.xml"))
cat(paste(c(l[1:41], "...", l[length(l) - 1]), collapse = "\n"))
```

# Generating general parameters XML files (param_gen.xml and param_newform.xml)

Nothing is to be done for specifying parameters values, the function is to be called with an optional argument for setting the desired version as follows.

<!-- Only for the show -->
```{r gen_gen_param, eval = FALSE}
# with the latest version
gen_general_param_xml(out_dir = workspace_path)

# or with a specific version
gen_general_param_xml(out_dir = workspace_path, stics_version = "V10.2.0")

```