Merge branch 'develop' into dist/risotto/risotto-2.8.0/develop

This attribute have 3 values:
- "no": it's a normal file
- "name": the name is included in another template so it is generated before other templates
- "content": the content is included in another template so it is generated before the other templates and is not installed on the server

fixes #2

README.md

@@ -0,0 +1,27 @@
+Rougail
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+[Documentation (french)](doc/README.md)

doc/README.md

@@ -0,0 +1,37 @@
+# Rougail
+
+Rougail est un bibliothèque python3 qui permet de charger des dictionnaires (fichiers au format XML), de charger les variables dans Tiramisu et de générer des templates.
+
+## La bibliothèque
+
+  - [Utiliser la bibliothèque](dev/README.md)
+  - [Personnaliser la configration de la bibliothèque](dev/config.md)
+
+## Les dictionnaires
+
+  - [Les dictionnaires](dictionary/rougail.md)
+  - [Les dictionnaires extra](dictionary/extra.md)
+
+### Les variables
+
+  - [Les familles](family/README.md)
+  - [Les variables](variable/README.md)
+
+### Les services
+
+  - [La gestion d'un fichier](service/file.md)
+  - [La gestion d'un fichier de service systemd](service/override.md)
+  - [La gestion d'une ip](service/ip.md)
+
+### Les contraintes
+
+  - [Les calcules automatiques](fill/README.md)
+  - [Les vérifications des valeurs](check/README.md)
+  - [Les conditions](condition/README.md)
+  - [Les variables meneuses ou suiveuses](variable/leadership.md)
+
+## Les templates
+
+  - Type creole
+  - Type jinja2
+FIXME ^^

doc/auto.rst

@@ -1,23 +0,0 @@
-Valeur automatiquement modifiée
-===============================
-
-Une variable avec valeur automatiquement modifiée est une variable dont la valeur sera considéré comme modifié quand le serveur sera déployé.
-
-Voici un variable a valeur automatiquement modifiée :
-
-    <variable name="my_variable" type="oui/non" description="My variable" auto_save="True">
-
-Dans ce cas la valeur est fixée à la valeur actuelle.
-Par exemple, si la valeur de cette variable est issue d'un calcul, la valeur ne sera plus recalculée.
-
-Valeur en lecture seule automatique
-===================================
-
-Une variable avec valeur en lecture seule automatique est une variable dont la valeur ne sera plus modifiable par l'utilisateur quand le serveur sera déployé.
-
-Voici un variable à valeur en lecture seule automatique :
-
-    <variable name="my_variable" type="oui/non" description="My variable" auto_freeze="True">
-
-Dans ce cas la valeur est fixée à la valeur actuelle et elle ne sera plus modifiable par l'utilisateur.
-Par exemple, si la valeur de cette variable est issue d'un calcul, la valeur ne sera plus recalculée.

doc/check/README.md

@@ -0,0 +1,5 @@
+# Les vérifications des valeurs
+
+  - [Fonction de vérification](function.md)
+  - [Les variables à choix](valid_enum.md)
+  - [Réfinition](redefine.md)

doc/check/function.md

@@ -0,0 +1,53 @@
+# Fonction de vérification
+
+## Vérification stricte des valeurs
+
+Une fonction de vérification est une fonction complémentaire au type qui permet de valider plus précisement le contenu d'une variable.
+
+Voici un exemple simple de validation des valeurs :
+
+```
+<variables>
+    <variable name="my_variable"/>
+</variables>
+<constraints>
+    <check name="islower">
+        <target>my_variable</target>
+    </check>
+</constraints>
+```
+
+La [cible (de type variable)](../target/variable.md) de la fonction de vérification est ici "my_variable".
+
+Dans cette exemple, la valeur de la variable "my_variable" va être validé par la fonction islower.
+
+Voici le contenu de la fonction :
+
+```
+def islower(value):
+    if value is None:
+        return
+    if not value.islower():
+        raise ValueError(f'"{value}" is not lowercase string')
+```
+
+Une fonction de vérification doit prendre en compte 2 aspects important :
+
+- la valeur peut ne pas être renseigné (même si la variable est obligatoire), la valeur None doit être prise en compte
+- si la valeur est invalide, il faut faire un raise de type ValueError avec, si possible, un message explicite.
+
+À partir de maintenant seule None et des valeurs en minuscule seront autorisés.
+
+Il est possible de définir des [paramètres](../param/README.md) à cette fonction.
+
+## Vérification des valeurs avec avertissement
+
+Dans la contrainte, il est possible de spécifier le niveau d'erreur et le mettre en avertissement :
+
+```
+<check name="islower" level="warning">
+    <target>my_variable</target>
+</check>
+```
+
+Dans ce cas une valeur avec une majuscule sera accepté, mais un message d'avertissement apparaitra.

doc/check/redefine.md

@@ -0,0 +1,61 @@
+# Rédéfinition
+
+## Redéfinition des vérification
+
+Dans un premier dictionnaire déclarons notre variable et sa fonction de vérification :
+
+```
+<variables>
+    <variable name="my_variable"/>
+</variables>
+<constraints>
+    <check name="islower">
+        <target>my_variable</target>
+    </check>
+</constraints>
+```
+
+Dans un second dictionnaire il est possible de redéfinir le calcul :
+
+```
+<variables>
+    <variable name="my_variable" redefine="True"/>
+</variables>
+<constraints>
+    <check name="isspace">
+        <target>my_variable</target>
+    </check>
+</constraints>
+```
+
+Dans ce cas, la fonction "islower" exécuté. Si cette fonction ne retourne pas d'erreur, la seconde fonction "isspace" sera exécuté.
+
+## Redéfinition avec suppression d'un calcul
+
+Il se peut que dans un dictionnaire on décide de vérifier la valeur d'une variable.
+
+Dans un second dictionnaire il est possible de supprimer cette vérification.
+
+Dans un premier dictionnaire déclarons notre variable et notre fonction de vérification :
+
+```
+<variables>
+    <variable name="my_variable"/>
+</variables>
+<constraints>
+    <check name="islower">
+        <target>my_variable</target>
+    </check>
+</constraints>
+```
+
+Dans un second dictionnaire supprimer cette vérification :
+
+```
+<variables>
+    <family name="family">
+        <variable name="my_variable" redefine="True" remove_check="True"/>
+    </family>
+</variables>
+```
+

doc/check/valid_enum.md

@@ -0,0 +1,46 @@
+# Les variables à choix
+
+Une variable à choix est d'abord une variable avec une [fonction check](function.md).
+
+## Les variables à choix simple
+
+Il est possible d'imposer une liste de valeur pour une variable particulière :
+
+```
+<check name="valid_enum">
+    <param>yes</param>
+    <param>no</param>
+    <param>maybe</param>
+    <target>my_variable</target>
+</check>
+```
+
+Dans ce cas, seule les valeurs proposés sont possible pour cette variable.
+
+Par défaut, cette variable est obligatoire. Cela signifie qu'il n'est pas possible de spécifier "None" à cette variable.
+
+## Les variables à choix avec valeur None
+
+Il y a deux possibilités pour avoir une valeur "None" dans les choix :
+
+- rendre la variable non obligatoire, cela va ajouter un choix "None" dans la liste :
+
+```
+<variable name="my_variable" mandatory="False">
+```
+
+Ou en ajoutant le paramètre "None" :
+
+```
+<check name="valid_enum">
+    <param>yes</param>
+    <param>no</param>
+    <param type='nil'/>
+    <param>maybe</param>
+    <target>my_variable</target>
+</check>
+```
+
+## La valeur par défaut
+
+Si aucune valeur n'est spécifié pour la variable, automatiquement le premier choix va est placé comme valeur par défaut.

doc/condition/README.md

@@ -0,0 +1,5 @@
+# Les conditions
+
+  - [Déclaration d'une condition](condition.md)
+  - [Les différentes conditions](conditions.md)
+  - [Réfinition](redefine.md)

doc/condition/condition.md

@@ -0,0 +1,76 @@
+# Les conditions
+
+## Un condition
+
+Les conditions permettent d'ajouter ou de supprimer des propriétés à une [variable](../variable/README.md), une [famille](../family/README.md), un [service](../service/service.md), un [fichier](../service/file.md) ou une [ip](../service/ip.md) suivant le contexte.
+
+Nous allons nous concentrer ici sur la condition hidden_if_in, mais [il existe d'autre conditions](conditions.md).
+
+La condition hidden_if_in permet de cacher une variable où une famille à l'utilisateur, mais cette variable est toujours accessible dans un calcul, un vérification ou dans un template.
+
+```
+<variables>
+    <variable name="condition" type="boolean"/>
+    <variable name="my_variable"/>
+</variables>
+
+<constraints>
+    <condition name="hidden_if_in" source="condition">
+        <param>True</param>
+        <target>my_variable</target>
+    </condition>
+</constraints>
+```
+
+Le [paramètres](../param/README.md) de la condition permet de définir les valeurs que doit avoir la source pour appliquer l'action.
+
+La [cible](../target/README.md) de la condition est ici "my_variable".
+
+Donc ici la variable est caché à l'utilisateur si la variable "condition" est à True (le paramètre).
+
+## Un condition avec plusieurs paramètres
+
+Il est également possible de mettre plusieurs paramètre :
+
+```
+<variables>
+    <variable name="condition"/>
+    <variable name="my_variable"/>
+</variables>
+
+<constraints>
+    <condition name="hidden_if_in" source="condition">
+        <param>yes</param>
+        <param>maybe</param>
+        <target>my_variable</target>
+    </condition>
+</constraints>
+```
+
+## Une condition optionnelle
+
+Il est possible de définir une condition avec une variable source qui n'existe pas dans toutes les contextes.
+
+Dans ce cas, on met la condition en "optionnelle".
+
+Si la variable source existe, la condition s'applique.
+
+Si la variable source n'existe pas :
+
+- si le nom fini en _if_in (par exemple hidden_if_in), l'action est forcée sans condition (les cibles sont hidden)
+- si le nom fini en _if_not_in (par exemple hidden_if_not_in), la condition est totalement ignorée
+
+Ces deux comportements peuvent être changé à tout moment avec l'attribut "apply_on_fallback". Dans ce cas :
+
+- si la valeur de "apply_on_fallback" est "True", l'action est forcée sans condition
+- si la valeur de "apply_on_fallback" est "False", la condition est totalement ignorée 
+
+Exemple :
+
+```
+<condition name="hidden_if_in" source="condition" optional="True", apply_on_fallback="False">
+    <param>yes</param>
+    <param>maybe</param>
+    <target>my_variable</target>
+</condition>
+```

doc/condition/conditions.md

@@ -0,0 +1,29 @@
+# Les conditions
+
+## Les conditions \_if_in et \_if_not_in
+
+Il existe deux types de conditions :
+
+- les conditions dont le nom fini par \_if_in : dans ce cas si la variable source est égal à un des paramètres, l'action est effective
+- les conditions dont le nom fini par \_if_not_in : dans ce cas si la variable source est différents de tous les paramètres, l'action est effective
+
+## Désactivation
+
+Il est possible de désactiver une [variable](../variable/README.md) ou une [famille](../family/README.md) avec les conditions :
+
+- disabled_if_in
+- disabled_if_not_in
+
+## Caché
+
+Il est possible de cacher une [variable](../variable/README.md), une [famille](../family/README.md), un [service](../service/service.md), un [fichier](../service/file.md) ou une [ip](../service/ip.md) avec les conditions :
+
+- hidden_if_in
+- hidden_if_not_in
+
+## Obligatoire
+
+Il est possible de rendre obligatoire une [variable](../variable/README.md) avec les conditions :
+
+- mandatory_if_in
+- mandatory_if_not_in

doc/condition/redefine.md

@@ -0,0 +1,29 @@
+# Rédéfinition
+
+Il se peut que dans un dictionnaire on décide de définir une condition.
+
+Dans un second dictionnaire il est possible de supprimer cette condition.
+
+Dans un premier dictionnaire déclarons notre variable et notre calcule :
+
+```
+<variables>
+    <variable name="condition" type="boolean"/>
+    <variable name="my_variable"/>
+</variables>
+
+<constraints>
+    <condition name="hidden_if_in" source="condition">
+        <param>True</param>
+        <target>my_variable</target>
+    </condition>
+</constraints>
+```
+
+Dans un second dictionnaire supprimer ce calcul :
+
+```
+<variables>
+    <variable name="condition" redefine="True" remove_condition="True"/>
+</variables>
+```

doc/dev/README.md

@@ -0,0 +1,191 @@
+# La bibliothèque rougail
+
+Rougail est une bibliothèque simple a utiliser.
+
+Dans les exemples suivant, nous utilisons la configuration par défaut de Rougail. Vous pouvez [personnaliser les répertoires utilisés](config.md).
+
+## Convertisons un dictionnaire en objet tiramisu
+
+Commençons par créer un dictionnaire simple.
+
+Voici un premier dictionnaire /srv/rougail/dictionaries/00-base.xml :
+
+```
+<?xml version='1.0' encoding='UTF-8'?>
+<rougail>
+    <variables>
+        <variable name="my_variable">
+            <value>my_value</value>
+        </variable>
+    </variables>
+</rougail>
+```
+
+Construisons les objets tiramisu :
+
+```python
+from rougail import RougailConvert
+
+rougail = RougailConvert()
+rougail.save('example.py')
+```
+
+Un nouveau fichier 'example.py' va être créé dans le répertoire local
+
+## Convertisons un dictionnaire extra en objet tiramisu
+
+En plus du dictionnaire précédent, créons un dictionnaire extra /srv/rougail/extra_dictionaries/00-base.xml
+
+```
+<?xml version='1.0' encoding='UTF-8'?>
+<rougail>
+    <variables>
+        <variable name="my_variable_extra">
+            <value>my_value_extra</value>
+        </variable>
+    </variables>
+</rougail>
+```
+
+Construisons les objets tiramisu :
+
+```python
+from rougail import RougailConvert, RougailConfig
+
+RougailConfig['extra_dictionaries']['example'] = ['/srv/rougail/extra_dictionaries/']
+
+rougail = RougailConvert()
+rougail.save('example.py')
+```
+
+## Templatisons un template
+
+Nous créons un dictionnaire complémentaire pour ajouter notre template /srv/rougail/dictionaries/00-template.xml :
+
+```
+<?xml version='1.0' encoding='UTF-8'?>
+<rougail>
+    <services>
+        <service name="example">
+            <file name="/etc/example.conf"/>
+        </service>
+    </services>
+</rougail>
+```
+
+Et un template /srv/rougail/templates/example.conf :
+
+```
+The value: %%my_variable
+
+The extra value: %%example.my_variable_extra
+```
+
+Générons le fichier tiramisu :
+
+```python
+from rougail import RougailConvert, RougailConfig
+
+RougailConfig['extra_dictionaries']['example'] = ['/srv/rougail/extra_dictionaries/']
+
+rougail = RougailConvert()
+rougail.save('example.py')
+```
+
+Créer les répertoires utils pour la templatisation :
+
+```bash
+mkdir /srv/rougail/destinations /srv/rougail/tmp
+```
+
+Générons le template :
+
+```python
+import asyncio
+from example import option_0
+from tiramisu import Config
+from rougail import RougailSystemdTemplate
+
+async def template():
+    config = await Config(option_0)
+    engine = RougailSystemdTemplate(config)
+    await engine.instance_files()
+
+loop = asyncio.get_event_loop()
+loop.run_until_complete(template())
+loop.close()
+```
+
+Le fichier /srv/rougail/destinations/etc/example.conf est maintenant créé avec le contenu attendu suivant :
+
+```
+The value: my_value
+
+The extra value: my_value_extra
+```
+
+## Appliquons un patch au template
+
+Il peut être intéressant de réaliser un patch à un template pour corriger un problème spécifique à notre environnement, sans attendre que le mainteneur du template n'est fait la correction.
+
+Testons en créant le patch /srv/rougail/patches/example.conf.patch :
+
+```
+--- /srv/rougail/templates/example.conf	2021-02-13 19:41:38.677491087 +0100
++++ tmp/example.conf	2021-02-13 20:12:55.525089820 +0100
+@@ -1,3 +1,5 @@
+ The value: %%my_variable
+ 
+ The extra value: %%example.my_variable_extra
++
++Add by a patch
+```
+
+Le patch est bien appliquer sur notre fichier /srv/rougail/destinations/etc/example.conf :
+
+```
+The value: my_value
+
+The extra value: my_value_extra
+
+Add by a patch
+```
+
+Deux choses importantes à savoir sur les patchs :
+
+- le nom du patch est obligatoire le nom du template source + ".patch"
+- la deuxième ligne doit toujours commencer par "+++ tmp/" + le nom du template source
+
+## Créons une fonction personnalisé
+
+Nous créons un dictionnaire complémentaire pour ajouter un calcul à la variable "my_variable" dans /srv/rougail/dictionaries/00-fill.xml :
+
+```
+<?xml version='1.0' encoding='UTF-8'?>
+<rougail>
+    <constraints>
+        <fill name="return_no">
+            <target>my_variable</target>
+        </fill>
+    </constraints>
+</rougail>
+```
+
+Puis créons la fonction "return_no" dans /srv/rougail/functions.py :
+
+```python
+def return_no():
+    return 'no'
+```
+
+Après avoir reconverti les dictionnaires et regénérer le template nous avons donc le contenu du fichier /srv/rougail/destinations/etc/example.conf :
+
+```
+The value: no
+
+The extra value: my_value_extra
+
+Add by a patch
+```
+
+La valeur de la variable "my_variable" est bien calculé à partir de la fonction "return_no".

doc/dev/config.md

@@ -0,0 +1,94 @@
+# Personnalisons la configuration de Rougail
+
+La configuration de rougail se trouve dans l'objet RougailConfig :
+
+```python
+from rougail import RougailConfig
+```
+
+C'est un simple dictionnaire python avec différentes clefs.
+
+Pour modifier il suffit de faire :
+
+```python
+RougailConfig[key] = value
+```
+
+## Les répertoires des dictionnaires
+
+Il existe deux types de répertoires de dictionnaires :
+
+- les dictionnaires principaux avec la clef "dictionaries_dir". La valeur par défaut est ['/srv/rougail/dictionaries']. Cette variable doit contenir la liste des répertoires contenants des dictionnaires.
+
+Les dictionnaires sont chargés dans l'ordre des répertoires. Chaque répertoire est chargé les uns après les autres. A l'intérieur de ces répertoires les fichiers XML seront classés par ordre alphabétique.
+
+Il n'y a pas de classement par ordre alphabétique de l'ensemble des fichiers XML de tous les répertoires.
+
+Les familles et variables de ces dictionnaires sont classés, par défaut, dans l'espace de nom "rougail". Il est possible de changer le nom de cet espace de nom avec la clef "variable_namespace".
+
+- les dictionnaires extra avec la clef "extra_dictionaries". La valeur est un dictionnaire avec l'ensemble des espaces de nom. La clef étant l'espace de nom et la valeur étant une liste de répertoire.
+
+Par exemple pour ajouter l'extra "example" il faut faire :
+
+```
+RougailConfig['extra_dictionaries']['example'] = ['/dir1', '/dir2']
+```
+
+Les dictionnaires sont chargés dans le même ordre que les dictionnaires principaux.
+
+## La DTD
+
+Rougail a besoin du fichier de DTD pour lire les fichiers dictionnaire.
+
+Par défaut le fichier de la DTD est dans le sous répertoire "data" du répertoire de code. Le nom du fichier est rougail.dtd.
+
+Pour pouvez changer le répertoire de destination de la DTD et le nom du fichier avec la clef "dtdfilename".
+
+## Le fichier de fonction
+
+Le fichier qui contient les fonctions personnalisés est géré dans la clef "functions_file" et a comme valeur par défaut "/srv/rougail/functions.py".
+
+## Le répertoire des templates
+
+Le répertoire des templates est géré dans la clef "templates_dir" et a comme valeur par défaut : "/srv/rougail/templates".
+
+## Le moteur de templates par défaut
+
+Le moteur de template est géré dans la clef "default_engine" et a comme valeur par défaut : "creole". Les valeurs possible sont "none", "creole" ou "jinja2".
+
+## Le répertoire des patchs
+
+Le répertoire des patches est géré dans la clef "patches_dir" et a comme valeur par défaut : "/srv/rougail/patches".
+
+## Le répertoire temporaire
+
+Le répertoire temporaire est utile lors de la génération de template. Il contient une copie des templates avec, éventuellement, les patches appliqués sur les templates.
+
+Le répertoire de temporaire est géré dans la clef "tmp_dir" et a comme valeur par défaut : "/srv/rougail/tmp".
+
+## Le répertoire de destination des fichiers générés
+
+Le répertoire de destination des fichiers générés est géré dans la clef "destinations_dir" et a comme valeur par défaut : "/srv/rougail/destinations".
+
+## La variable auto_freeze
+
+La propriété auto_freeze n'est appliqué que une variable spécifique passe à True. Par défaut le nom de la variable est "instancied_module", mais il est possible de changer le nom de cette variable via la clef "auto_freeze_variable".
+
+## Les modes
+
+Les modes sont personnalisables dans Rougail. Par défaut les modes sont "basic", "normal" et "expert".
+Il est possible de changer cette liste via la clef "modes_level".
+
+Si vous changer ces valeurs, penser à changer les modes par défaut des familles et des variables.
+
+## Le mode par défaut pour une famille
+
+Le mode par défaut d'une famille est "basic". Il est possible de changer le mode par défaut d'une famille via la clef "default_family_mode".
+
+## Le mode par défaut pour une variable
+
+Le mode par défaut d'une variable est "normal". Il est possible de changer le mode par défaut d'une variable via la clef "default_variable_mode".
+
+## Le nom des fonctions internes
+
+Il est possible d'ajouter des fonctions interne via la clef "internal_functions".

doc/dictionary/extra.md

@@ -0,0 +1,12 @@
+# Les dictionnaires extra
+
+Un extra est un espace de nom différent. L'idée et de pouvoir classer les variables par thématique.
+
+Les espaces de nom extra doivent être déclaré au moment [de la configuration de Rougail](../dev/config.md).
+
+Dans cet espace de nom :
+
+- des variables et des familles peuvent avoir le même nom dans différentes familles
+- la valeur d'un cible, source, leader ou follower des contraintes doivent être avec un chemin complet
+- on ne peut pas déclarer des services dans cet espace de nom
+- dans un template il faut utiliser des chemins complet (%%my_extra.my_family.my_variable ou %%my_extra.my_family.leader.follower pour une variable suiveuse)

doc/dictionary/rougail.md

@@ -0,0 +1,22 @@
+# Les dictionnaires
+
+## Un dictionnaire ?
+
+Un dictionnaire est un fichier XML donc la structure est décrite dans cette documentation.
+
+Un dictionnaire contient en ensemble de variable chargé dans Tiramisu, utilisable à tout moment, notamment dans des templates.
+
+Les familles, les variables et les contraintes peuvent être défini dans plusieurs dictionnaires. Ces dictionnaires s'aggrège alors.
+
+Il est également possible de redéfinir des éléments pour changer les comportement d'une variable ou d'un service.
+
+## L'espace de nom par défaut
+
+L'espace de nom par défaut s'appelle "rougail" ([ce nom est personnalisable](../dev/config.md)).
+
+Cet espace de nom est un peu particulier :
+
+- le nom des variables et des familles doivent être unique pour l'ensemble de cet espace (même si ces variables ou familles sont dans des familles différentes)
+- la valeur d'un cible, source, leader ou follower des contraintes peuvent être avec nom de la variable ou de la famille ou leurs chemins complet
+- on peut déclarer des services dans cet espace de nom
+- dans un template on peut utiliser cette variable sans le chemin complet (%%my_variable) ou avec (%%rougail.my_family.my_variable)

doc/family/README.md

@@ -0,0 +1,5 @@
+# Famille
+
+  - [Une famille](simple.md)
+  - [Famille crée dynamiquement](auto.md)
+

doc/family/auto.md

@@ -0,0 +1,27 @@
+# Famille crée dynamiquement
+
+Pour créer une famille dynamiquement, il faut créer une famille fictive lié à une variable.
+Le nom et la description de la famille et des variables qu'elle contient sera en réalité le prefix du nouveau nom/description. Le suffix viendra de la variable liée.
+
+Par exemple :
+
+```
+<variable name='varname' multi="True">
+    <value>val1</value>
+    <value>val2</value>
+</variable>
+<family name="my_dyn_family_" dynamic="varname" description="Describe ">
+    <variable name="my_dyn_var_"/>
+</family>
+```
+
+Créera deux familles :
+
+- la famille dynamique : "my_dyn_family_val1" avec la description "Describe val1"
+- la famille dynamique : "my_dyn_family_val2" avec la description "Describe val2"
+
+Dans la famille dynamique "my_dyn_family_val1" on retrouvera une variable "my_dyn_var_val1".
+
+Bien évidement si le contenu de "varname" venait a évolué, de nouvelles familles dynamiques pouvent apparaitre ou des familles dynamiques peuvent disparaître.
+
+Attention la variable lié à la famille doit être obligatoirement une variable multiple et il n'est pas possible de mettre une famille dans une famille dynamique.

doc/family/simple.md

@@ -0,0 +1,68 @@
+# Une famille
+
+Une famille est un conteneur de variables. Elle peut contenir également des familles.
+
+Pour décrire une famille il faut mettre au minimum un nom :
+
+```
+<family name="my_family"/>
+```
+
+Cette famille doit être placé dans une balise [variables](../variables.md) :
+
+```
+<variables>
+    <family name="my_family"/>
+</variables>
+```
+
+Ou dans une autre famille :
+
+```
+<variables>
+    <family name="my_family">
+        <family name="second_family"/>
+    </family>
+</variables>
+```
+
+Attention, une famille vide sera automatiquement supprimée.
+
+## Description et aide de la famille
+
+En plus d'un nom, il est possible de mettre une "description" à la famille. C'est une information "utilisateur" qui nous permettra d'avoir des informations complémentaires sur le contenu de cette famille :
+
+```
+<family name="my_family" description="This is a great family"/>
+```
+
+En plus de la description, il est possible de préciser une aide complémentaire :
+
+```
+<family name="my_family" help="This is a great family"/>
+```
+
+## Mode de la famille
+
+Le [mode](../mode.md) par défaut d'une famille correspond au [mode](../mode.md) du mode le plus petit entre la famille parente, les variables enfants ou des familles enfants qui sont contenu dans cette famille.
+
+Changer le [mode](../mode.md) d'une famille permet de définir le [mode](../mode.md) par défaut des variables ou des familles inclusent dans cette famille.
+
+Pour définir le [mode](../mode.md) :
+
+```
+<family name="my_family" mode="expert"/>
+```
+
+## Famille invisible
+
+Il est possible de cacher une famille, ainsi que toutes les variables et des familles inclusent dans cette famille.
+
+Cacher une famille signifie qu'elle ne sera pas visible lorsqu'on modifie la configuration du service.
+Par contre ces variables sont accessibles lorsqu'on va utiliser ces variables.
+
+Pour cacher une famille :
+
+```
+<family name="my_family" hidden="True"/>
+```

doc/fill.rst

@@ -1,285 +0,0 @@
-Les variables calculées
-=======================
-
-Une variable calculée est une variable donc sa valeur est le résultat d'une fonction python.
-
-Variable avec une valeur par défaut calculée
---------------------------------------------
-
-Créons une variable de type "oui/non" donc la valeur est retournée par la fonction "return_no" :
-
-    <variables>
-        <family name="family">
-            <variable name="my_calculated_variable" type="oui/non" description="My calculated variable"/>
-        </family>
-    </variables>
-    <constraints>
-        <fill name="return_no" target="my_calculated_variable"/>
-    </constraints>
-
-Puis créons la fonction "return_no" :
-
-    def return_no():
-        return 'non'
-
-Dans ce cas, la valeur par défaut est la valeur retournée par la fonction (ici "non"), elle sera calculée tant que l'utilisateur n'a pas de spécifié une valeur à cette variable.
-
-Si l'utilisateur à définit une valeur par défaut à "my_calculated_variable" :
-
-    <variable name="my_calculated_variable" type="oui/non" description="My calculated variable">
-        <value>oui</value>
-    </variable>
-
-Cette valeur par défaut sera complètement ignorée.
-
-Variable avec une valeur calculée
----------------------------------
-
-En ajoutant le paramètre "hidden" à "True" dans la variable précédente, l'utilisateur n'aura plus la possibilité de modifié la valeur. La valeur de la variable sera donc systématiquement calculée :
-
-    <variable name="my_calculated_variable" type="oui/non" description="My calculated variable" hidden="True"/>
-
-Si une condition "hidden_if_in" est spécifié à la variable, la valeur sera modifiable par l'utilisateur si elle n'est pas cachée mais elle sera systèmatiquement calculée (même si elle a déjà était modifiée) si la variable est cachée.
-
-Variable avec valeur calculée obligatoire
------------------------------------------
-
-Par défaut les variables calculées ne sont pas des varibles obligatoires.
-Dans ce cas un calcul peut retourner None, mais surtout un utilisateur peut spécifier une valeur nulle à cette variable. Dans ce cas le calcul ne sera pas réalisé.
-
-Fonction avec une valeur fixe comme paramètre positionnel
----------------------------------------------------------
-
-Déclarons un calcul avec paramètre :
-
-    <constraints>
-        <fill name="return_value" target="my_calculated_variable">
-            <param>non</param>
-        </fill>
-    </constraints>
-
-Créons la fonction correspondante :
-
-    def return_value(value):
-        return value
-
-La variable aura donc "non" comme valeur puisque le paramètre aura la valeur fixe "non".
-
-Paramètre nommée
-----------------
-
-Déclarons une contrainte avec un paramètre nommée :
-
-    <constraints>
-        <fill name="return_value" target="my_calculated_variable">
-            <param name="valeur">non</param>
-        </fill>
-    </constraints>
-
-Dans ce cas la fonction return_value sera exécuté avec le paramètre nommé "valeur" dont sa valeur sera "non".
-
-Paramètre avec un nombre
-------------------------
-
-Déclarons un calcul avec paramètre avec un nombre :
-
-    <constraints>
-        <fill name="return_value_with_number" target="my_calculated_variable">
-            <param type="number">1</param>
-        </fill>
-    </constraints>
-
-Créons la fonction correspondante :
-
-    def return_value_with_number(value):
-        if value == 1:
-            return 'non'
-        return 'oui'
-
-La variable aura donc "non" comme valeur puisque le paramètre aura la valeur fixe "1".
-
-Paramètre dont la valeur est issue d'une autre variable
--------------------------------------------------------
-
-Créons deux variables avec une contrainte de type variable qui contient le nom de la variable dont sa valeur sera utilisé comme paramètre :
-
-    <variables>
-        <family name="family">
-            <variable name="my_calculated_variable" type="oui/non" description="My calculated variable"/>
-            <variable name="my_variable" type="number" description="My variable">
-                <value>1</value>
-            </variable>
-        </family>
-    </variables>
-    <constraints>
-        <fill name="return_value_with_number" target="my_calculated_variable">
-            <param type="variable">my_variable</param>
-        </fill>
-    </constraints>
-
-Si l'utilisateur laisse la valeur 1 à "my_variable", la valeur par défault de la variable "my_calculated_variable" sera "non".
-Si la valeur de "my_variable" est différent de 1, la valeur par défaut de la variable "my_calculated_variable" sera "oui".
-
-Paramètre dont la valeur est issue d'une information de la configuration
-------------------------------------------------------------------------
-
-Créons une variable et la contrainte :
-
-    <variables>
-        <family name="family">
-            <variable name="my_calculated_variable" type="string" description="My calculated variable"/>
-        </family>
-    </variables>
-    <constraints>
-        <fill name="return_value" target="my_calculated_variable">
-            <param type="information">server_name</param>
-        </fill>
-    </constraints>
-
-Dans ce cas, l'information de la configuration "server_name" sera utilisé comme valeur de la variable "my_calculated_variable".
-Si l'information n'existe pas, la paramètre aura la valeur "None".
-
-Paramètre avec variable potentiellement non existante
------------------------------------------------------
-
-Suivant le contexte une variable peut exister ou ne pas exister.
-
-Un paramètre de type "variable" peut être "optional" :
-
-    <variables>
-        <family name="family">
-            <variable name="my_calculated_variable" type="oui/non" description="My calculated variable"/>
-        </family>
-    </variables>
-    <constraints>
-        <fill name="return_value" target="my_calculated_variable">
-            <param type="variable" optional="True">unknow_variable</param>
-        </fill>
-    </constraints>
-
-Dans ce cas la fonction "return_value" est exécuté sans paramètre.
-
-Paramètre avec variable potentiellement désactivée
---------------------------------------------------
-
-FIXME :
-
-<!ATTLIST param notraisepropertyerror (True|False) "False">
-
-Il n'y a pas spécialement de test !
-
-Les variables suiveuses
------------------------
-
-FIXME : 
-
-- tests/flattener_dicos/10leadership_append/00-base.xml
-- tests/flattener_dicos/10leadership_auto/00-base.xml
-- tests/flattener_dicos/10leadership_autoleader/00-base.xml
-- tests/flattener_dicos/10leadership_autoleader_expert/00-base.xml
-
-Les variables dynamiques
-------------------------
-
-Paramètre avec variable dynamique
-'''''''''''''''''''''''''''''''''
-
-Il est possible de faire un calcul avec comme paramètre une variable dynamique mais pour une suffix particulier :
-
-    <variables>
-        <family name='family'>
-            <variable name='suffixes' type='string' description="Suffixes of dynamic family" multi="True">
-                <value>val1</value>
-                <value>val2</value>
-            </variable>
-            <variable name="my_calculated_variable" type="string" description="My calculated variable"/>
-        </family>
-        <family name='dyn' dynamic="suffixes">
-            <variable name='vardyn' type='string' description="Dynamic variable">
-                <value>val</value>
-            </variable>
-        </family>
-    </variables>
-    <constraints>
-        <fill name="return_value" target="my_calculated_variable">
-            <param type="variable">vardynval1</param>
-        </fill>
-    </constraints>
-
-Dans ce cas, valeur du paramètre de la fonction "return_value" sera la valeur de la variable "vardyn" avec le suffix "val1".
-
-Calcule d'une variable dynamique
-''''''''''''''''''''''''''''''''
-
-Il est également possible de calculer une variable dynamique à partir d'une variable standard :
-
-    <variables>
-        <family name='family'>
-            <variable name='suffixes' type='string' description="Suffixes of dynamic family" multi="True">
-                <value>val1</value>
-                <value>val2</value>
-            </variable>
-            <variable name="my_variable" type="string" description="My variable">
-                <value>val</value>
-            </variable>
-        </family>
-        <family name='dyn' dynamic="suffixes">
-            <variable name="my_calculated_variable_dyn_" type="string" description="My calculated variable"/>
-                <value>val</value>
-            </variable>
-        </family>
-    </variables>
-    <constraints>
-        <fill name="return_value" target="my_calculated_variable_dyn_">
-            <param type="variable">my_variable</param>
-        </fill>
-    </constraints>
-
-Dans ce cas, les variables dynamiques "my_calculated_variable_dyn_" seront calculés à partir de la valeur de la variable "my_variable".
-Que cela soit pour la variable "my_calculated_variable_dyn_val1" et "my_calculated_variable_dyn_val2".
-
-Par contre, il n'est pas possible de faire un calcul pour une seule des deux variables issues de la variable dynamique.
-Si c'est ce que vous cherchez à faire, il faudra prévoir un traitement particulier dans votre fonction.
-
-Dans ce cas, il faut explicitement demander la valeur du suffix dans la fonction :
-
-    <constraints>
-        <fill name="return_value_suffix" target="my_calculated_variable_dyn_">
-            <param type="variable">my_variable</param>
-            <param type="suffix"/>
-        </fill>
-    </constraints>
-
-Et ainsi faire un traitement spécifique pour ce suffix :
-
-    def return_value_suffix(value, suffix):
-        if suffix == 'val1':
-            return value
-
-Redéfinition des calcules
--------------------------
-
-Dans un premier dictionnaire déclarons notre variable et notre calcule :
-
-    <variables>
-        <family name="family">
-            <variable name="my_calculated_variable" type="oui/non" description="My calculated variable"/>
-        </family>
-    </variables>
-    <constraints>
-        <fill name="return_no" target="my_calculated_variable"/>
-    </constraints>
-
-Dans un second dictionnaire il est possible de redéfinir le calcul :
-
-    <variables>
-        <family name="family">
-            <variable name="my_calculated_variable" redefine="True"/>
-        </family>
-    </variables>
-    <constraints>
-        <fill name="return_yes" target="my_calculated_variable"/>
-    </constraints>
-
-Dans ce cas, à aucun moment la fonction "return_no" ne sera exécuté. Seul la fonction "return_yes" le sera.
-

doc/fill/README.md

@@ -0,0 +1,6 @@
+# Les variables calculées
+
+Une variable calculée est une variable donc sa valeur est le résultat d'une fonction python.
+
+  - [Valeur calculée de la variable](value.md)
+  - [Réfinition](redefine.md)

doc/fill/redefine.md

@@ -0,0 +1,58 @@
+# Rédéfinition
+
+## Redéfinition des calcules
+
+Dans un premier dictionnaire déclarons notre variable et notre calcule :
+
+```
+<variables>
+    <variable name="my_calculated_variable"/>
+</variables>
+<constraints>
+    <fill name="return_no">
+        <target>my_calculated_variable</target>
+    </fill>
+</constraints>
+```
+
+Dans un second dictionnaire il est possible de redéfinir le calcul :
+
+```
+<variables>
+    <variable name="my_calculated_variable" redefine="True"/>
+</variables>
+<constraints>
+    <fill name="return_yes">
+        <target>my_calculated_variable</target>
+    </fill>
+</constraints>
+```
+
+Dans ce cas, à aucun moment la fonction "return_no" ne sera exécuté. Seul la fonction "return_yes" le sera.
+
+## Redéfinition avec suppression d'un calcul
+
+Il se peut que dans un dictionnaire on décide de définir une valeur par défaut à une variable via un calcul.
+
+Dans un second dictionnaire il est possible de supprimer ce calcul.
+
+Dans un premier dictionnaire déclarons notre variable et notre calcule :
+
+```
+<variables>
+    <variable name="my_calculated_variable"/>
+</variables>
+<constraints>
+    <fill name="return_no">
+        <target>my_calculated_variable"</target>
+    </fill>
+</constraints>
+```
+
+Dans un second dictionnaire supprimer ce calcul :
+
+```
+<variables>
+    <variable name="my_calculated_variable" redefine="True" remove_fill="True"/>
+</variables>
+```

doc/fill/value.md

@@ -0,0 +1,112 @@
+# Valeur calculée de la variable
+
+## Variable avec une valeur par défaut calculée
+
+Créons une variable dont la valeur est retournée par la fonction "return_no" :
+
+```
+<variables>
+    <variable name="my_calculated_variable"/>
+</variables>
+<constraints>
+    <fill name="return_no">
+        <target>my_calculated_variable</target>
+    </fill>
+</constraints>
+```
+
+Puis créons la fonction "return_no" :
+
+```
+def return_no():
+    return 'no'
+```
+
+La [cible (de type variable)](../target/variable.md) du calcul est ici "my_calculated_variable".
+
+Dans ce cas, la valeur par défaut est la valeur retournée par la fonction (ici "no"), elle sera calculée tant que l'utilisateur n'a pas de spécifié de valeur à cette variable.
+
+Attention, si une valeur par défaut est définit dans la variable "my_calculated_variable" :
+
+```
+<variable name="my_calculated_variable">
+    <value>yes</value>
+</variable>
+```
+Cette valeur par défaut sera complètement ignorée. C'est le calcul qui en définira la valeur.
+
+Il est possible de définir des [paramètres](../param/README.md) à cette fonction.
+
+## Variable avec une valeur calculée
+
+En ajoutant le paramètre "hidden" à "True" dans la variable précédente, l'utilisateur n'aura plus la possibilité de modifié la valeur. La valeur de la variable sera donc systématiquement calculée :
+
+```
+<variable name="my_calculated_variable" hidden="True"/>
+```
+
+Si une condition "hidden_if_in" est spécifié à la variable, la valeur sera modifiable par l'utilisateur si elle n'est pas cachée mais elle sera systèmatiquement calculée (même si elle a déjà était modifiée) si la variable est cachée.
+
+## Variable meneuse ou suiveuse avec valeur calculé
+
+Une [variable suiveuse](../variable/leadership.md) ne peut pas être calculé automatiquement.
+Une [variable meneuse](../variable/leadership.md) peut être calculé automatiquement.
+Si la variable n'est pas multiple, il ne faut pas que le calcule retourne une liste.
+
+## Variable dynamique avec une valeur calculée
+
+Il est également possible de calculer [une variable d'une famille dynamique](../family/auto.md) à partir d'une variable standard :
+
+```
+<variables>
+    <variable name='suffixes' type='string' description="Suffixes of dynamic family" multi="True">
+        <value>val1</value>
+        <value>val2</value>
+    </variable>
+    <variable name="my_variable" type="string" description="My variable">
+        <value>val</value>
+    </variable>
+    <family name='dyn' dynamic="suffixes">
+        <variable name="my_calculated_variable_dyn\_" type="string" description="My calculated variable"/>
+            <value>val</value>
+        </variable>
+    </family>
+</variables>
+<constraints>
+    <fill name="return_value">
+        <param type="variable">my_variable</param>
+        <target>my_calculated_variable_dyn_</target>
+    </fill>
+</constraints>
+```
+
+Dans ce cas, les variables dynamiques "my_calculated_variable_dyn_" seront calculés à partir de la valeur de la variable "my_variable".
+Que cela soit pour la variable "my_calculated_variable_dyn_val1" et "my_calculated_variable_dyn_val2".
+
+Par contre, il n'est pas possible de faire un calcul pour une seule des deux variables issues de la variable dynamique.
+Si c'est ce que vous cherchez à faire, il faudra prévoir un traitement particulier dans votre fonction.
+
+Dans ce cas, il faut explicitement demander la valeur du suffix dans la fonction :
+
+```
+<constraints>
+    <fill name="return_value_suffix">
+        <param type="variable">my_variable</param>
+        <param type="suffix"/>
+        <target>my_calculated_variable_dyn_</target>
+    </fill>
+</constraints>
+```
+
+Et ainsi faire un traitement spécifique pour ce suffix :
+
+```
+def return_value_suffix(value, suffix):
+    if suffix == 'val1':
+        return value
+```
+
+## Variable avec valeur calculée obligatoire
+
+Par défaut les variables calculées ne sont pas des variables obligatoires.
+Dans ce cas un calcul peut retourner "None" ou "", mais surtout un utilisateur peut spécifier une valeur nulle à cette variable. Dans ce cas le calcul ne sera plus réalisé.

doc/mode.md

@@ -0,0 +1,10 @@
+Mode
+====
+
+Par défault, il existe trois "mode" dans Rougail :
+
+- basic : variables indispensables à la mise en place d'un service
+- normal : variables couramment modifié par l'utilisateur
+- expert : variables a manipuler avec précausion et en toute connaissance de cause
+
+Il est possible de personnaliser les modes dans la [configuration de rougail](dev/config.md)

doc/param/README.md

@@ -0,0 +1,7 @@
+# Paramètre de la fonction
+
+  - [Paramètre positionnel ou nommée](positional.md)
+  - [Type de paramètre simple](simple.md)
+  - [Type de paramètre "variable"](variable.md)
+  - [Type de paramètre "information"](information.md)
+

doc/param/information.md

@@ -0,0 +1,24 @@
+# Paramètre de type information
+
+## Les informations de la configuration
+
+Le paramètre peut être la valeur est issue d'une information de la configuration :
+
+```
+<param type="information">server_name</param>
+```
+
+Dans ce cas, l'information de la configuration "server_name" sera utilisé comme valeur du paramètre.
+Si l'information n'existe pas, la paramètre aura la valeur "None".
+
+## Les informations de la cible
+
+Le paramètre peut être la valeur est issue d'une information de la cible du calcul :
+
+```
+<param type="target_information">test</param>
+<param type="target_information">help</param>
+```
+
+Dans ce cas, l'information de la configuration "test" ou "help" sera utilisé comme valeur du paramètre.
+Si l'information n'existe pas, la paramètre aura la valeur "None".

doc/param/positional.md

@@ -0,0 +1,26 @@
+# Paramètre positionnel
+
+Déclarons un paramètre positionnel :
+
+```
+<param>no</param>
+```
+
+Créons la fonction correspondante :
+
+```
+def return_value(value):
+    return value
+```
+
+La variable "value" de la fonction "return_value" aura donc "no" comme valeur puisque le paramètre aura la valeur fixe "no".
+
+# Paramètre nommée
+
+Déclarons maintenant un paramètre nommée :
+
+```
+<param name="valeur">no</param>
+```
+
+Dans ce cas la fonction return_value sera exécuté avec le paramètre nommé "valeur" dont sa valeur sera "no".

doc/param/simple.md

@@ -0,0 +1,44 @@
+# Paramètre de type "texte"
+
+Déclarons un paramètre avec une string :
+
+```
+<param type="string">no</param>
+```
+
+C'est le type par défaut pour un paramètre.
+
+# Paramètre de type "nombre"
+
+Déclarons un paramètre avec un nombre :
+
+```
+<param type="number">1</param>
+```
+
+Créons la fonction correspondante :
+
+```
+def return_value_with_number(value):
+    if value == 1:
+        return 'no'
+    return 'yes'
+```
+
+La variable aura donc "no" comme valeur puisque le paramètre aura la valeur fixe "1".
+
+# Paramètre de type "booléen"
+
+Déclarons un paramètre avec un booléen :
+
+```
+<param type="boolean">True</param>
+```
+
+# Paramètre de type "nil"
+
+Le paramètre peut être une valeur null (None en python) :
+
+```
+<param type="nil"/>
+```

doc/param/variable.md

@@ -0,0 +1,55 @@
+# Paramètre de type "variable"
+
+Imaginons que la variable "my_variable" pré-existe. La valeur de la variable sera utilisé comme paramètre :
+
+```
+<param type="variable">my_variable</param>
+```
+
+[Les variables meneuses ou suiveuses](../variable/leadership.md) peuvent être utilisé sans soucis comme paramètre.
+
+## Paramètre avec variable potentiellement non existante
+
+Suivant le contexte une variable peut exister ou ne pas exister.
+
+Un paramètre de type "variable" peut être "optional" :
+
+```
+<param type="variable" optional="True">unknow_variable</param>
+```
+
+Si la variable "unknow_variable" n'existe pas, le paramètre ne sera pas passé à la fonction.
+
+Si maintenant on créé un nouveau dictionnaire en créant cette variable, la fonction sera exécuté avec le paramètre.
+
+## Paramètre avec variable potentiellement désactivée
+
+Si une variable est désactivé, l'utilisation de cette variable peut poser problème.
+
+Il est possible de ne pas générer d'erreur si une variable est désactivé en utilisant le paramètre "propertyerror" :
+
+```
+<param type="variable" propertyerror="False">variable1</param>
+```
+
+Dans ce cas, si la variable est désactivé, le paramètre n'est jamais donnée à la fonction de destination.
+
+## Paramètre avec variable dynamique
+
+Il est possible de faire un calcul avec comme paramètre [une variable d'une famille dynamique](../family/auto.md) mais pour une suffix particulier.
+
+Par exemple :
+
+```
+<param type="variable">vardynval1</param>
+```
+
+Dans ce cas, la valeur du paramètre de la fonction sera la valeur de la variable "vardyn" pour la famille ayant le suffix "val1".
+
+Il peut être utile de récupérer la valeur du suffix dans la fonction, pour cela il suffit de mettre un paramètre de type suffix :
+
+```
+<param type="suffix"/>
+```
+
+Dans l'exemple précédent la valeur de ce paramètre sera "val1".

doc/service/README.md

@@ -0,0 +1,4 @@
+# Les services
+
+
+

doc/service/file.md

@@ -0,0 +1,168 @@
+# La gestion d'un fichier
+
+## La balise file
+
+La gestion des fichiers se fait dans un conteneur de [service](service.md).
+
+La déclaration du fichier permet de générer un fichier à partir d'un template pour le déposer à l'endroit prévu dans la déclaration de cette élément.
+
+Il est nécessaire, au minimum, de spécifier le chemin complet du fichier :
+
+```
+<services>
+    <service name="squid">
+        <file>/etc/squid/squid.conf</file>
+    </service>
+</services>
+```
+
+Dans ce cas, le nom du template est déduit du nom du fichier, ici cela sera "squid.conf".
+
+Si le template a un nom différent (par exemple si plusieurs template se retrouve avec le même nom), il est possible de changer le nom du template avec l'attribut source :
+
+```
+<file source="template-squid.conf">/etc/squid/squid.conf</file>
+```
+
+## Les noms de fichiers dynamique
+
+Il est possible également de définir le nom du fichier dans une variable :
+
+```
+<services>
+    <service name="squid">
+        <file file_type="variable" source="squid.conf">my_variable</file>
+    </service>
+</services>
+<variables>
+    <variable name="my_variable">
+        <value>/etc/squid/squid.conf</value>
+    </variable>
+</variables>
+```
+
+Attention, la variable doit être de type "filename".
+
+Dans le cas des fichiers dynamique, la source est obligatoire.
+
+Il est même possible de définir une variable de type multiple, ce qui génèrera plusiers fichiers :
+
+```
+<services>
+    <service name="squid">
+        <file file_type="variable" source="squid.conf">my_variable</file>
+    </service>
+</services>
+<variables>
+    <variable name="my_variable" multi="True">
+        <value>/etc/squid1/squid.conf</value>
+        <value>/etc/squid2/squid.conf</value>
+    </variable>
+</variables>
+```
+
+Dans ce cas là, le fichier source est identique mais les fichiers de destination seront différent.
+
+Il peut être important de personnaliser le contenu du fichier suivant le fichier de destination.
+Dans ce cas il y a deux possibilités :
+
+- la variable "rougail_filename" contient le nom de fichier de destination
+- l'utilisateur de l'attribut "variable" 
+
+En effet, il est possible de passer le contenu d'une variable au template :
+
+```
+<services>
+    <service name="squid">
+        <file file_type="variable" source="squid.conf" variable="my_variable2">my_variable1</file>
+    </service>
+</services>
+<variables>
+    <variable name="my_variable1" multi="True">
+        <value>/etc/squid1/squid.conf</value>
+        <value>/etc/squid2/squid.conf</value>
+    </variable>
+    <variable name="my_variable2" multi="True">
+        <value>squid1</value>
+        <value>squid2</value>
+    </variable>
+</variables>
+```
+
+Dans ce cas, lors de la génération du fichier /etc/squid1/squid.conf on retrouvera la variable "rougail_variable" avec la valeur "squid1" et le fichier /etc/squid2/squid.conf on retrouvera la variable "rougail_variable" avec la valeur "squid2".
+
+Attention : les deux variables "my_variable1" et "my_variable2" doivent être multiple et de même longueur.
+
+## Les droits des fichiers
+
+Par défaut les droits du fichier généré sont "0644" avec comme utilisateur "root" et groupe "root".
+
+Il est possible de définir une autre valeur à un ou plusieurs de ces attributs :
+
+```
+<file mode="0640" owner="nobody" group="squid">/etc/squid/squid.conf</file>
+```
+
+## Désactiver la génération d'un fichier
+
+Il est possible de définir une [condition](../condition/README.md) de type "disabled_if_in" ou "disabled_if_not_in" sur une balise fichier :
+
+```
+<services>
+    <service name="test">
+        <file filelist="squid">/etc/squid/squid.conf</file>
+    </service>
+</services>
+<variables>
+    <variable name="condition" type="boolean"/>
+</variables>
+<constraints>
+    <condition name="disabled_if_in" source="condition">
+        <param>False</param>
+        <target type="filelist">squid</target>
+    </condition>
+</constraints>
+```
+
+Dans ce cas, tous les fichiers avec un attribut filelist à "squid" seront désactivés si la variable "condition" est False.
+
+## Redéfinir une fichier
+
+Il est possible de redéfinir les éléments d'un fichier dans un dictionnaire différent en utilisant l'attribut redefine :
+
+```
+<file source="template-squid.conf" redefine="True">/etc/squid/squid.conf</file>
+```
+
+## Choix du moteur de templating
+
+Par défaut, le moteur de templating est le moteur de templating compatible avec "creole".
+
+Il est possible de désactiver la templatisation du fichier (il sera alors uniquement copié) :
+
+```
+<file engine="none">/etc/squid/squid.conf</file>
+```
+
+Ou d'utiliser le moteur "jinja2" :
+
+```
+<file engine="jinja2">/etc/squid/squid.conf</file>
+```
+
+## Inclusion de template
+
+Un attribut "included" permet de définir la nature du fichier. Cet attribut peut avoir trois valeurs :
+
+- "no" : c'est un fichier normal
+- "name" : le répertoire de destination est listé dans un autre template, il faut que le fichier soit généré avant cet autre template
+- "content" : le contenu du fichier est copié dans un autre template, il faut que le fichier soit généré avant cet autre template et ce fichier n'a pas besoin d'être installé sur le serveur cible.
+
+Exemples :
+
+```
+<file included="name">/etc/squid/squid.conf</file>
+<file included="content">/etc/squid/squid.conf</file>
+```
+
+Bien entendu, c'est au développeur de lister ou d'inclure le contenu de ce template dans le fichier de destination. Cet attribut permet juste de garantir que le fichier sera fait avant l'autre et de ne pas l'installer sur le serveur si ce n'est pas nécessaire.

doc/service/ip.md

@@ -0,0 +1,47 @@
+# La gestion d'une IP
+
+## La balise IP
+
+La gestion des IP se fait dans un conteneur de [service](service.md).
+
+La déclaration de l'attribut permet d'associer une IP autorisé à accéder au service.
+
+Il est nécessaire, au minimum, de spécifier le nom d'une variable de type "IP" :
+
+```
+<ip ip_type="variable">variable_ip</ip>
+```
+
+## La gestion d'un réseau
+
+L'adresse peut être de type réseau ("network") :
+
+```
+<ip netmask="variable_netmask">variable_ip</ip>
+```
+
+Attention, dans ce cas il faut préciser une variable de type "netmask" dans l'attribut netmask.
+
+## Désactiver la génération d'une IP
+
+Il est possible de définir une [condition](../condition/README.md) de type "disabled_if_in" ou "disabled_if_not_in" sur une balise IP :
+
+```
+<services>
+    <service name="test">
+        <ip iplist="test_ip">variable_ip</ip>
+    </service>
+</services>
+<variables>
+    <variable name="condition" type="boolean"/>
+    <variable name="variable_ip" type="ip"/>
+</variables>
+<constraints>
+    <condition name="disabled_if_in" source="condition">
+        <param>False</param>
+        <target type="iplist">test_ip</target>
+    </condition>
+</constraints>
+```
+
+Dans ce cas, tous les IP avec un attribut iplist à "test_ip" seront désactivé si la variable "condition" est False.

doc/service/override.md

@@ -0,0 +1,43 @@
+# Override
+
+## La balise override
+
+La gestion des overrides se fait dans un conteneur de [service](service.md).
+
+La balise override permet de redéfinir facilement un service systemd.
+
+Il suffit d'avoir un template dont le nom est par défaut le nom du service avec l'extension "service" et de déclarer la balise :
+
+```
+<services>
+    <service name="squid">
+        <override/>
+    </service>
+</services>
+```
+
+Dans cette exemple, le template associé doit s'appeler squid.service
+
+Si le fichier service a un nom différent (par exemple si plusieurs template se retrouve avec le même nom), il est possible de changer le nom du template avec l'attribut source :
+
+```
+<override source="test.service"/>
+```
+
+Dans ce cas le fichier de destination aura le même nom.
+
+## Choix du moteur de templating
+
+Par défaut, le moteur de templating est le moteur de templating compatible avec "creole".
+
+Il est possible de désactiver la templatisation du fichier (il sera alors uniquement copié) :
+
+```
+<override engine="none"/>
+```
+
+Ou d'utiliser le moteur "jinja2" :
+
+```
+<override engine="jinja2"/>
+```

doc/service/service.md

@@ -0,0 +1,50 @@
+# La gestion d'un service
+
+## La base service
+
+Un service est inclut dans un conteneur [services](../services.md).
+
+Cette balise permet de définir tous les éléments ([fichier](file.md), [IP](ip.md), ...) liés à un service ou à démon.
+
+Il faut, à la création du service, préciser son nom :
+
+```
+<?xml version='1.0' encoding='UTF-8'?>
+<rougail>
+    <services>
+        <service name="squid"/>
+    </services>
+</rougail>
+```
+
+Un service peut ne pas être géré :
+
+```
+<service name="squid" manage="True"/>
+```
+
+Un service non géré est généralement une service qui n'existe pas réellement (par exemple si on configure un client).
+
+Un service non géré ne peut conteneur que des fichiers.
+
+## Désactiver la génération d'un service
+
+Il est possible de définir une [condition](../condition/README.md) de type "disabled_if_in" ou "disabled_if_not_in" sur une balise service :
+
+```
+<services>
+    <service name="test">
+    </service>
+</services>
+<variables>
+    <variable name="condition" type="boolean"/>
+</variables>
+<constraints>
+    <condition name="disabled_if_in" source="condition">
+        <param>False</param>
+        <target type="servicelist">test</target>
+    </condition>
+</constraints>
+```
+
+Dans ce cas, tous les services et les éléments qu'il compose ([fichier](file.md), ...) avec un attribut servicelist à "test" seront désactivés si la variable "condition" est False.

doc/services.md

@@ -0,0 +1,14 @@
+# Le conteneur des services
+
+La balise "services" est le conteneur de l'ensemble des [services](service/service.md).
+
+Il est placé à la racine du dictionnaire :
+
+```
+<?xml version='1.0' encoding='UTF-8'?>
+<rougail>
+    <services/>
+</rougail>
+```
+
+Attention, cette balise ne peut pas être placé dans un dictionnaire "extra".

doc/target/README.md

@@ -0,0 +1,5 @@
+# La cible
+
+  - [De type variable](../target/variable.md)
+  - [De type famille](../target/family.md)
+  - [De type \*list](../target/list.md)

doc/target/family.md

@@ -0,0 +1,13 @@
+# Cible de type "variable"
+
+Une cible peut être de type famille :
+
+```
+<target type="family">my_family</target>
+```
+
+Mais une target peut être optionnelle. C'est à dire que si la famille n'existe pas, l'action ne sera pas associé à cette famille.
+
+```
+<target type="family" optional="True">my_family</target>
+```

doc/target/list.md

@@ -0,0 +1,35 @@
+# Cible de type \*list
+
+## Les différences cible de type \*.list
+
+### servicelist
+
+Une cible peut être de type [service](../service/service.md) :
+
+```
+<target type="servicelist">example</target>
+```
+
+### filelist
+
+Une cible peut être de type [fichier](../service/file.md) :
+
+```
+<target type="filelist">example</target>
+```
+
+### iplist
+
+Une cible peut être de type [ip](../service/ip.md) :
+
+```
+<target type="iplist">example</target>
+```
+
+## La cible optionnelle
+
+Mais une target peut être optionnelle. C'est à dire que si la \*list n'existe pas, l'action ne sera pas associé.
+
+```
+<target type="filelist" optional="True">unknown</target>
+```

doc/target/variable.md

@@ -0,0 +1,13 @@
+# Cible de type "variable"
+
+Par défaut une cible est de type variable.
+
+```
+<target>my_variable</target>
+```
+
+Mais une target peut être optionnelle. C'est à dire que si la variable n'existe pas, l'action ne sera pas associé à cette variable.
+
+```
+<target optional='True'>my_variable</target>
+```

doc/variable.rst

@@ -1,13 +0,0 @@
-Variable
-========
-
-Variable obligatoire
---------------------
-
-Variable dont une valeur est requise :
-
-    <variables>
-        <family name="family">
-            <variable name="my_variable" type="oui/non" description="My variable" mandatory="True"/>
-        </family>
-    </variables>

doc/variable/README.md

@@ -0,0 +1,4 @@
+# Variable
+
+  - [Une variable](simple.md)
+  - [Variable meneuse ou suiveuse](leadership.md)

doc/variable/leadership.md

@@ -0,0 +1,102 @@
+# Variable meneuse ou suiveuse
+
+## Variable meneuse
+
+Une variable meneuse est une variable qui va guider la longueur d'autre variables (appelé variables suiveuse).
+
+Une variable meneuse est une [variable](simple.md) qui est obligatoirement de type multiple.
+
+Une variable meneuse peut être obligatoire.
+
+Le [mode](../mode.md) par défaut correspond au plus petit mode définit par l'utilisateur des variables suiveuses.
+
+## Variable suiveuse
+
+Une variable suiveuse est une variable donc la longueur n'est pas déterminé par elle-même, mais est identique à celle de la variable meneuse dont elle dépend.
+
+Une variable suiveuse est une variable placer juste derrière une variable meneuse ou une autre variable suiveuse.
+
+L'ordre de définition des variables suiveuses est important.
+
+Cette variable peut être de type multiple. Dans ce cas, pour un index determiné, il est possible de mettre plusieurs valeurs à une même variable.
+
+Une variable suiveuse peut être obligatoire. Cela signifie que lorsqu'une variable meneuse est renseigné, il faut obligatoirement que la variable suiveuse est également une valeur à l'index considéré.
+Si aucune valeur n'est définit pour la variable meneuse, aucune valeur n'est a spécifié pour la variable suiveuse.
+
+Le [mode](../mode.md) par défaut d'une variable suiveuse correspond au [mode](../mode.md) de la variable meneuse.
+
+Si une variable meneuse est caché ou désactivé, les variables suiveuses le seront également.
+
+## Définition des variables meneuse et suiveuse
+
+Voici un exemple de définition d'une variable meneuse et de deux variables meneuses :
+
+```
+<variables>
+    <family name="family">
+        <variable name="leader" multi='True'/>
+        <variable name="follower1"/>
+        <variable name="follower2" multi='True'/>
+    </family>
+</variables>
+<constraints>
+    <group leader="leader">
+        <follower>follower1</follower>
+        <follower>follower2</follower>
+    </group>
+</constraints>
+```
+
+Le nom et la description du groupe prendra par défaut le nom et la description de la variable meneuse.
+
+Il est possible d'en définit d'autres :
+
+```
+<constraints>
+    <group leader="leader" name="leadership" description="My leadership">
+        <follower>follower1</follower>
+        <follower>follower2</follower>
+    </group>
+</constraints>
+```
+
+## Définition des variables meneuse et suiveuse dans un dictionnaire extra
+
+Voici un exemple de définition d'une variable meneuse et de deux variables meneuses dans un espace de nom "example" :
+
+```
+<variables>
+    <family name="family">
+        <variable name="leader" multi='True'/>
+        <variable name="follower1"/>
+        <variable name="follower2" multi='True'/>
+    </family>
+</variables>
+<constraints>
+    <group leader="example.family.leader">
+        <follower>example.family.follower1</follower>
+        <follower>example.family.follower2</follower>
+    </group>
+</constraints>
+```
+
+Le chemin de la variable meneuse est traditionnel, par contre le chemin des variables suiveuses n'est pas le chemin définitif de la variable.
+
+Le chemin d'une variable suiveuse est normalement "example.family.leader.follower1" mais la variable n'est pas encore une variable suiveuse à ce moment là du traitement. C'est pour cela qu'il ne faut, uniquement dans les groupes, mettre le nom de la variable meneuse dans le chemin.
+
+## Ajout d'une nouvelle variable suiveuse
+
+Pour ajouter, dans un nouveau dictionnaire, une variable suiveuse à notre groupe, rien de plus simple, il suffit de redéfinir un groupe avec cette (ou ces) nouvelle variable :
+
+```
+<variables>
+    <family name="family">
+        <variable name="follower3"/>
+    </family>
+</variables>
+<constraints>
+    <group leader="leader">
+        <follower>follower3</follower>
+    </group>
+</constraints>
+```

doc/variable/simple.md

@@ -0,0 +1,255 @@
+# Variable
+
+## Un variable
+
+Une variable est forcement dans [variables](../variables.md) ou dans une [famille](../family/README.md).
+
+Une variable est déjà un nom. C'est à dire qu'on pourra utiliser plus tard la variable via ce nom.
+
+```
+<variables>
+    <variable name="my_variable"/>
+    <family name="my_family">
+        <variable name="my_family_variable"/>
+    </variable>
+</variables>
+```
+
+## Description et aide sur la variable
+
+En plus d'un nom, il est possible de mettre une "description" à la variable. C'est une information "utilisateur" qui nous permettra d'avoir des informations complémentaires sur le contenu de cette variable :
+
+```
+<variable name="my_variable" description="This is a greate variable"/>
+```
+
+En plus de la description, il est possible de préciser une aide complémentaire :
+
+```
+<variable name="my_variable" help="This is a greate variable"/>
+```
+
+Cette aide peut être utilisé à tout moment comme valeur [d'un paramètre](../param/information.md).
+
+## Le type de la variable
+
+Une variable a un type. Ce type permet de définir les valeurs acceptées par cette variable :
+
+- string : chaine de caractère (type par défaut)
+- number : un nombre
+- float : un chiffre flottant
+- boolean : "True" ou "False", si aucune valeur n'est défini la valeur par défaut de cette variable sera "True", ces variables sont également obligatoire par défaut
+- password : un mot de passe
+- mail : une adresse mail
+- filename : nom de fichier au sens Unix (exemple : "/etc/passwd")
+- date : une date au format "%Y-%m-%d" (exemple : "2021-01-30")
+- unix_user : nom d'utilisateur au sens Unix
+- ip : n'importe quelle adresse IPv4
+- cidr : n'importe quelle adresse IPv4 au format CIDR
+- local_ip : adresse IPv4 sur un réseau local, si l'adresse IPv4 n'est pas local, un warning sera afficher mais la valeur sera accepté tout de même
+- netmask : masque d'une adresse IPv4
+- network : adresse réseau
+- network_cidr : adresse réseau au format CIDR
+- broadcast : adresse de diffusion
+- netbios : nom netbios
+- domain : nom de domaine
+- hostname : nom d'hôte
+- web_address : adresse web (http://www.cadoles.com/)
+- port : port
+- mac : adresse MAC
+- schedule : périodicité du schedule, les valeurs possibles sont "none", "daily", "weekly" ou "monthly"
+- schedulemod : type de schedule, les valeurs possibles sont "pre" ou "post"
+
+Pour définir le type d'une variable :
+
+```
+<variable name="my_variable" type="number"/>
+```
+
+## Variable à valeur multiple
+
+Par défaut une variable ne peut acceuillir qu'une seule valeur. Il peut être utile de pouvoir spécifier plusieurs valeurs à une même variable.
+
+Pour définir une variable à valeur multiple :
+
+```
+<variable name="my_variable" multi="True"/>
+```
+
+## Variable invisible
+
+Il est possible de cacher une variable.
+
+Cacher une variable signifie qu'elle ne sera pas visible lorsqu'on modifie la configuration du service.
+Par contre cette variable sera accessibles lorsqu'on va l'utiliser.
+
+Pour cacher une variable :
+
+```
+<variable name="my_variable" hidden="True"/>
+```
+
+## Variable désactive
+
+Il est possible de désactiver une variable.
+
+Désactiver une variable signifie qu'elle ne sera pas visible lorsqu'on modifie la configuration du service mais également lorsqu'on va l'utiliser.
+
+Pour désactiver une variable :
+
+```
+<variable name="my_variable" disabled="True"/>
+```
+
+## Variable obligatoire
+
+Variable dont une valeur est requise :
+
+```
+<variable name="my_variable" mandatory="True"/>
+```
+
+Les variables booléans sont par défaut obligatoire. Pour qu'une variable booléan ne soit pas obligatoire il faut le préciser explicitement :
+
+```
+<variable name="my_variable" type="boolean" mandatory="False"/>
+```
+
+Les variables avec une valeur par défaut (non calculée) sont également automatiquement obligatoire.
+[Les variables à choix](../check/valid_enum.md) sans choix "None" sont également automatiquement obligatoire.
+
+## Valeur par défaut d'une variable
+
+Il est possible de fixer les valeurs par défaut d'une variable :
+
+```
+<variable name="my_variable">
+    <value>value</value>
+</variable>
+```
+
+Pour une variable multiple, il est possible de préciser plusieurs valeurs :
+
+```
+<variable name="my_variable" multi="True">
+    <value>value 1</value>
+    <value>value 2</value>
+</variable>
+```
+
+Si la variable n'est pas pas une [variable meneuse](leadership.md), la première valeur défini dans cette liste sera également la valeur par défaut proposé si on ajoute une nouvelle valeur à cette variable.
+
+Une valeur par défaut peut également être [une valeur calculer](../fill/README.md).
+
+## Redéfinir une variable
+
+Il est possible de définir une variable dans un dictionnaire et de changer son comportement dans une second dictionnaire.
+
+Attention trois attributs ne sont redéfinisable :
+
+- name
+- type
+- multi
+
+Créons notre variable :
+
+    <variable name="my_variable"/>
+
+Et redéfinisons là :
+
+```
+<variable name="my_variable" redefine="True" description="New description"/>
+```
+
+## Créer une variable inexistante
+
+Il est parfois utile de créer une variable si elle n'existe pas dans un autre dictionnaire :
+
+```
+<variable name="my_variable" exists="False"/>
+```
+
+Si cette variable existe dans un autre dictionnaire, elle ne sera pas modifié ni recréé
+
+## Redéfinir une variable si elle existe
+
+Parfois on veut pouvoir redéfinir une variable mais seulement dans le cas où elle existe déjà :
+
+```
+<variable name="my_variable" redefine="True" exists="True" hidden="True"/>
+```
+
+## Variable à valeur automatiquement modifiée
+
+Une variable avec valeur automatiquement modifiée est une variable dont la valeur sera considéré comme modifié quand la propriété global "force_store_value" de Tiramisu est mise.
+
+Voici une variable a valeur automatiquement modifiée :
+
+```
+<variable name="my_variable" auto_save="True">
+    <value>my_value</value>
+</variable>
+```
+
+Dans ce cas la valeur est fixée à la valeur actuelle.
+Par exemple, si la valeur de cette variable est issue d'un calcul, la valeur ne sera plus recalculée.
+
+Ces variables sont généralement des variables obligatoires. En effet ces variable ne sont automatiquement modifiées que si elles ont une valeurs.
+
+Une [variable meneuse ou suiveuse](leadership.md) ne peut pas avoir la propriété auto_save.
+
+## Variable à valeur en lecture seule automatique
+
+Une variable avec valeur en lecture seule automatique est une variable dont la valeur ne sera plus modifiable par l'utilisateur quand la [variable "server_deployed" passe à "True"](../dev/config.md).
+
+Voici un variable à valeur en lecture seule automatique :
+
+```
+<variable name="server_deployed" type="boolean">
+    <value>False</value>
+</variable>
+<variable name="my_variable" auto_freeze="True"/>
+```
+
+Dans ce cas la valeur est fixée à la valeur actuelle et elle ne sera plus modifiable par l'utilisateur.
+Par exemple, si la valeur de cette variable est issue d'un calcul, la valeur ne sera plus recalculée.
+
+Ces variables sont généralement des variables obligatoires. En effet ces variable ne sont en lecteur seul que si elles ont une valeurs.
+
+Une [variable meneuse ou suiveuse](leadership.md) ne peut pas avoir la propriété auto_freeze.
+
+## Information "test"
+
+L'attribut "test" est un attribut spécial qui permet aux concepteurs d'un dictionnaire d'influancer le robot de test en précisant de valeurs utile à tester.
+
+Concrêtement, le contenu de cet attribut est enregister dans une "information" de l'option Tiramisu correspondante.
+
+Exemple :
+
+```
+<variable name="my_variable" test="yes"/>
+```
+
+Il est possible de préciser plusieurs valeurs avec le séparateur "|" :
+
+```
+<variable name="my_variable" test="yes|no"/>
+```
+
+Cette valeur peut être utilisé à tout moment comme valeur [d'un paramètre](../param/information.md).
+
+## Mode de la variable
+
+Le [mode](../mode.md) par défaut d'une variable correspond au [mode](../mode.md) de la [famille](../family/README.md).
+
+Cas particuliers :
+
+- une variable à valeur automatiquement modifiée ou une variable en lecture seule automatique est par défaut en mode "basic".
+- si la variable n'est pas dans une famille, la variable aura le mode "normal" par défaut.
+- une variable obligatoire sans valeur par défaut (calculer ou non) aura le mode "basic".
+
+Pour définir le [mode](../mode.md) :
+
+```
+<variable name="my_variable" mode="expert"/>
+```

doc/variables.md

@@ -0,0 +1,12 @@
+# Le conteneur des variables
+
+La balise "variables" est le conteneur de l'ensemble des [familles](family/README.md) et des [variables](variable/README.md).
+
+Il est placé à la racine du dictionnaire :
+
+```
+<?xml version='1.0' encoding='UTF-8'?>
+<rougail>
+    <variables/>
+</rougail>
+```

src/rougail/__init__.py

@@ -1,5 +1,32 @@
-#from .loader import load
+"""Rougail method
-from .objspace import CreoleObjSpace
-from .annotator import modes
 
-__ALL__ = ('CreoleObjSpace', 'modes')
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from .convert import RougailConvert
+from .template.systemd import RougailSystemdTemplate
+from .config import RougailConfig
+from rougail.update import RougailUpgrade
+
+__ALL__ = ('RougailConvert', 'RougailSystemdTemplate', 'RougailConfig', 'RougailUpgrade')

src/rougail/annotator.py

@@ -1,1293 +0,0 @@
-# coding: utf-8
-from copy import copy
-from typing import List
-
-from collections import OrderedDict
-from os.path import join, basename
-from ast import literal_eval
-import imp
-
-
-from .i18n import _
-from .utils import normalize_family
-from .error import DictConsistencyError
-
-#mode order is important
-modes_level = ('basic', 'normal', 'expert')
-
-
-class Mode(object):
-    def __init__(self, name, level):
-        self.name = name
-        self.level = level
-    def __gt__(self, other):
-        return other.level < self.level
-
-
-def mode_factory():
-    mode_obj = {}
-    for idx in range(len(modes_level)):
-        name = modes_level[idx]
-        mode_obj[name] = Mode(name, idx)
-    return mode_obj
-
-modes = mode_factory()
-
-
-CONVERT_OPTION = {'number': dict(opttype="IntOption", func=int),
-                  'float': dict(opttype="FloatOption", func=float),
-                  'choice': dict(opttype="ChoiceOption"),
-                  'string': dict(opttype="StrOption"),
-                  'password': dict(opttype="PasswordOption"),
-                  'mail': dict(opttype="EmailOption"),
-                  'boolean': dict(opttype="BoolOption"),
-                  'symlink': dict(opttype="SymLinkOption"),
-                  'filename': dict(opttype="FilenameOption"),
-                  'date': dict(opttype="DateOption"),
-                  'unix_user': dict(opttype="UsernameOption"),
-                  'ip': dict(opttype="IPOption", initkwargs={'allow_reserved': True}),
-                  'local_ip': dict(opttype="IPOption", initkwargs={'private_only': True, 'warnings_only': True}),
-                  'netmask': dict(opttype="NetmaskOption"),
-                  'network': dict(opttype="NetworkOption"),
-                  'broadcast': dict(opttype="BroadcastOption"),
-                  'netbios': dict(opttype="DomainnameOption", initkwargs={'type': 'netbios', 'warnings_only': True}),
-                  'domain': dict(opttype="DomainnameOption", initkwargs={'type': 'domainname', 'allow_ip': False}),
-                  'hostname': dict(opttype="DomainnameOption", initkwargs={'type': 'hostname', 'allow_ip': False}),
-                  'web_address': dict(opttype="URLOption", initkwargs={'allow_ip': True, 'allow_without_dot': True}),
-                  'port': dict(opttype="PortOption", initkwargs={'allow_private': True}),
-                  'mac': dict(opttype="MACOption"),
-                  'cidr': dict(opttype="IPOption", initkwargs={'cidr': True}),
-                  'network_cidr': dict(opttype="NetworkOption", initkwargs={'cidr': True}),
-                  }
-
-
-# a CreoleObjSpace's attribute has some annotations
-# that shall not be present in the exported (flatened) XML
-ERASED_ATTRIBUTES = ('redefine', 'exists', 'fallback', 'optional', 'remove_check', 'namespace',
-                     'remove_condition', 'path', 'instance_mode', 'index', 'is_in_leadership',
-                     'level', 'remove_fill', 'xmlfiles')
-ERASED_CONTAINER_ATTRIBUTES = ('id', 'container', 'group_id', 'group', 'container_group')
-
-FORCE_CHOICE = {'oui/non': ['oui', 'non'],
-                'on/off': ['on', 'off'],
-                'yes/no': ['yes', 'no'],
-                'schedule': ['none', 'daily', 'weekly', 'monthly'],
-                'schedulemod': ['pre', 'post']}
-
-KEY_TYPE = {'variable': 'symlink',
-            'SymLinkOption': 'symlink',
-            'PortOption': 'port',
-            'UnicodeOption': 'string',
-            'NetworkOption': 'network',
-            'NetmaskOption': 'netmask',
-            'URLOption': 'web_address',
-            'FilenameOption': 'filename'}
-
-FREEZE_AUTOFREEZE_VARIABLE = 'module_instancie'
-
-PROPERTIES = ('hidden', 'frozen', 'auto_freeze', 'auto_save', 'force_default_on_freeze',
-              'force_store_value', 'disabled', 'mandatory')
-CONVERT_PROPERTIES = {'auto_save': ['force_store_value'], 'auto_freeze': ['force_store_value', 'auto_freeze']}
-
-RENAME_ATTIBUTES = {'description': 'doc'}
-INTERNAL_FUNCTIONS = ['valid_enum', 'valid_in_network', 'valid_differ', 'valid_entier']
-
-class SpaceAnnotator:
-    """Transformations applied on a CreoleObjSpace instance
-    """
-    def __init__(self, objectspace, eosfunc_file):
-        self.objectspace = objectspace
-        GroupAnnotator(objectspace)
-        ServiceAnnotator(objectspace)
-        VariableAnnotator(objectspace)
-        ConstraintAnnotator(objectspace,
-                            eosfunc_file,
-                            )
-        FamilyAnnotator(objectspace)
-        PropertyAnnotator(objectspace)
-
-
-class GroupAnnotator:
-    def __init__(self,
-                 objectspace,
-                 ):
-        self.objectspace = objectspace
-        if not hasattr(self.objectspace.space, 'constraints') or not hasattr(self.objectspace.space.constraints, 'group'):
-            return
-        self.convert_groups()
-
-    def convert_groups(self):  # pylint: disable=C0111
-        for group in self.objectspace.space.constraints.group:
-            leader_fullname = group.leader
-            leader_family_name = self.objectspace.paths.get_variable_family_name(leader_fullname)
-            leader_name = self.objectspace.paths.get_variable_name(leader_fullname)
-            namespace = self.objectspace.paths.get_variable_namespace(leader_fullname)
-            if '.' not in leader_fullname:
-                leader_fullname = '.'.join([namespace, leader_family_name, leader_fullname])
-            follower_names = list(group.follower.keys())
-            has_a_leader = False
-            ori_leader_family = self.objectspace.paths.get_family_obj(leader_fullname.rsplit('.', 1)[0])
-            for variable in list(ori_leader_family.variable.values()):
-                if has_a_leader:
-                    # it's a follower
-                    self.manage_follower(namespace,
-                                         leader_family_name,
-                                         variable,
-                                         leadership_name,
-                                         follower_names,
-                                         leader_space,
-                                         leader_is_hidden,
-                                         )
-                    ori_leader_family.variable.pop(variable.name)
-                    if follower_names == []:
-                        # no more follower
-                        break
-                elif variable.name == leader_name:
-                    # it's a leader
-                    if isinstance(variable, self.objectspace.Leadership):
-                        # append follower to an existed leadership
-                        leader_space = variable
-                        # if variable.hidden:
-                        #     leader_is_hidden = True
-                    else:
-                        leader_space = self.objectspace.Leadership(variable.xmlfiles)
-                        if hasattr(group, 'name'):
-                            leadership_name = group.name
-                        else:
-                            leadership_name = leader_name
-                        leader_is_hidden = self.manage_leader(leader_space,
-                                                              leader_family_name,
-                                                              leadership_name,
-                                                              leader_name,
-                                                              namespace,
-                                                              variable,
-                                                              group,
-                                                              leader_fullname,
-                                                              )
-                    has_a_leader = True
-            else:
-                raise DictConsistencyError(_('cannot found followers {}').format(follower_names))
-        del self.objectspace.space.constraints.group
-
-    def manage_leader(self,
-                      leader_space: 'Leadership',
-                      leader_family_name: str,
-                      leadership_name: str,
-                      leader_name: str,
-                      namespace: str,
-                      variable: 'Variable',
-                      group: 'Group',
-                      leader_fullname: str,
-                      ) -> None:
-        # manage leader's variable
-        if variable.multi is not True:
-            raise DictConsistencyError(_('the variable {} in a group must be multi').format(variable.name))
-        leader_space.variable = []
-        leader_space.name = leadership_name
-        leader_space.hidden = variable.hidden
-        if variable.hidden:
-            leader_is_hidden = True
-            variable.frozen = True
-            variable.force_default_on_freeze = True
-        else:
-            leader_is_hidden = False
-        variable.hidden = None
-        if hasattr(group, 'description'):
-            leader_space.doc = group.description
-        elif hasattr(variable, 'description'):
-            leader_space.doc = variable.description
-        else:
-            leader_space.doc = variable.name
-        leadership_path = namespace + '.' + leader_family_name + '.' + leadership_name
-        self.objectspace.paths.add_family(namespace,
-                                          leadership_path,
-                                          leader_space,
-                                          )
-        leader_family = self.objectspace.space.variables[namespace].family[leader_family_name]
-        leader_family.variable[leader_name] = leader_space
-        leader_space.variable.append(variable)
-        self.objectspace.paths.set_leader(namespace,
-                                          leader_family_name,
-                                          leader_name,
-                                          leader_name,
-                                          )
-        leader_space.path = leader_fullname
-        return leader_is_hidden
-
-    def manage_follower(self,
-                        namespace: str,
-                        leader_family_name: str,
-                        variable: 'Variable',
-                        leader_name: str,
-                        follower_names: List[str],
-                        leader_space: 'Leadership',
-                        leader_is_hidden: bool,
-                        ) -> None:
-        if variable.name != follower_names[0]:
-            raise DictConsistencyError(_('cannot found this follower {}').format(follower_names[0]))
-        follower_names.remove(variable.name)
-        if leader_is_hidden:
-            variable.frozen = True
-            variable.force_default_on_freeze = True
-        leader_space.variable.append(variable)  # pylint: disable=E1101
-        self.objectspace.paths.set_leader(namespace,
-                                          leader_family_name,
-                                          variable.name,
-                                          leader_name,
-                                          )
-
-
-class ServiceAnnotator:
-    """Manage service's object
-    for example::
-        <services>
-            <service name="test">
-                <service_access service='ntp'>
-                    <port protocol='udp' service_accesslist='ntp_udp'>123</port>
-                </service_access>
-            </service>
-        </services>
-    """
-    def __init__(self, objectspace):
-        self.objectspace = objectspace
-        self.convert_services()
-
-    def convert_services(self):
-        if not hasattr(self.objectspace.space, 'services'):
-            return
-        if not hasattr(self.objectspace.space.services, 'service'):
-            del self.objectspace.space.services
-            return
-        self.objectspace.space.services.hidden = True
-        self.objectspace.space.services.name = 'services'
-        self.objectspace.space.services.doc = 'services'
-        families = {}
-        for idx, service_name in enumerate(self.objectspace.space.services.service.keys()):
-            service = self.objectspace.space.services.service[service_name]
-            new_service = self.objectspace.service(service.xmlfiles)
-            for elttype, values in vars(service).items():
-                if not isinstance(values, (dict, list)) or elttype in ERASED_ATTRIBUTES:
-                    setattr(new_service, elttype, values)
-                    continue
-                eltname = elttype + 's'
-                path = '.'.join(['services', service_name, eltname])
-                family = self.gen_family(eltname,
-                                         path,
-                                         service.xmlfiles,
-                                         )
-                if isinstance(values, dict):
-                    values = list(values.values())
-                family.family = self.make_group_from_elts(service_name,
-                                                          elttype,
-                                                          values,
-                                                          path,
-                                                          )
-                setattr(new_service, elttype, family)
-            new_service.doc = new_service.name
-            families[service_name] = new_service
-        self.objectspace.space.services.service = families
-
-    def gen_family(self,
-                   name,
-                   path,
-                   xmlfiles
-                   ):
-        family = self.objectspace.family(xmlfiles)
-        family.name = normalize_family(name)
-        family.doc = name
-        family.mode = None
-        self.objectspace.paths.add_family('services',
-                                          path,
-                                          family,
-                                          )
-        return family
-
-    def make_group_from_elts(self,
-                             service_name,
-                             name,
-                             elts,
-                             path,
-                             ):
-        """Splits each objects into a group (and `OptionDescription`, in tiramisu terms)
-        and build elements and its attributes (the `Options` in tiramisu terms)
-        """
-        families = []
-        new_elts = self._reorder_elts(name,
-                                      elts,
-                                      )
-        for index, elt_info in enumerate(new_elts):
-            elt = elt_info['elt']
-            elt_name = elt_info['elt_name']
-
-            # try to launch _update_xxxx() function
-            update_elt = '_update_' + elt_name
-            if hasattr(self, update_elt):
-                getattr(self, update_elt)(elt,
-                                          index,
-                                          path,
-                                          service_name,
-                                          )
-
-            idx = 0
-            while True:
-                if hasattr(elt, 'source'):
-                    c_name = elt.source
-                else:
-                    c_name = elt.name
-                if idx:
-                    c_name += f'_{idx}'
-                subpath = '{}.{}'.format(path, c_name)
-                if not self.objectspace.paths.family_is_defined(subpath):
-                    break
-                idx += 1
-            family = self.gen_family(c_name,
-                                     subpath,
-                                     elt.xmlfiles,
-                                     )
-            family.variable = []
-            listname = '{}list'.format(name)
-            activate_path = '.'.join([subpath, 'activate'])
-            for key in dir(elt):
-                if key.startswith('_') or key.endswith('_type') or key in ERASED_ATTRIBUTES:
-                    continue
-                value = getattr(elt, key)
-                if key == listname:
-                    self.objectspace.list_conditions.setdefault(listname,
-                                                                {}).setdefault(
-                                                                    value,
-                                                                    []).append(activate_path)
-                    continue
-                family.variable.append(self._generate_element(elt_name,
-                                                              key,
-                                                              value,
-                                                              elt,
-                                                              f'{subpath}.{key}'
-                                                              ))
-            # FIXME ne devrait pas etre True par défaut
-            # devrait etre un calcule
-            family.variable.append(self._generate_element(elt_name,
-                                                         'activate',
-                                                          True,
-                                                          elt,
-                                                          activate_path,
-                                                          ))
-            families.append(family)
-        return families
-
-    def _generate_element(self,
-                          elt_name,
-                          key,
-                          value,
-                          elt,
-                          path,
-                          ):
-        variable = self.objectspace.variable(elt.xmlfiles)
-        variable.name = normalize_family(key)
-        variable.mode = None
-        if key == 'name':
-            true_key = elt_name
-        else:
-            true_key = key
-        dtd_key_type = true_key + '_type'
-        if key == 'activate':
-            type_ = 'boolean'
-        elif hasattr(elt, dtd_key_type):
-            type_ = KEY_TYPE[getattr(elt, dtd_key_type)]
-        elif key in self.objectspace.booleans_attributs:
-            type_ = 'boolean'
-        else:
-            type_ = 'string'
-        variable.type = type_
-        if type_ == 'symlink':
-            variable.opt = self.objectspace.paths.get_variable_path(value,
-                                                                    'services',
-                                                                    )
-#            variable.opt = value
-            variable.multi = None
-        else:
-            variable.doc = key
-            val = self.objectspace.value(elt.xmlfiles)
-            val.type = type_
-            val.name = value
-            variable.value = [val]
-        self.objectspace.paths.add_variable('services',
-                                            path,
-                                            'service',
-                                            False,
-                                            variable,
-                                            )
-        return variable
-
-    def _reorder_elts(self,
-                      name,
-                      elts,
-                      ):
-        """Reorders by index the elts
-        """
-        new_elts = {}
-        # reorder elts by index
-        for idx, elt in enumerate(elts):
-            new_elts.setdefault(idx, []).append(elt)
-        idxes = list(new_elts.keys())
-        idxes.sort()
-        result_elts = []
-        for idx in idxes:
-            for elt in new_elts[idx]:
-                result_elts.append({'elt_name': name, 'elt': elt})
-        return result_elts
-
-    def _update_override(self,
-                         file_,
-                         index,
-                         service_path,
-                         service_name,
-                         ):
-        file_.name = f'/systemd/system/{service_name}.service.d/rougail.conf'
-        # retrieve default value from File object
-        for attr in ['owner', 'group', 'mode']:
-            setattr(file_, attr, getattr(self.objectspace.file, attr))
-        if not hasattr(file_, 'source'):
-            file_.source = f'{service_name}.service'
-        self._update_file(file_,
-                          index,
-                          service_path,
-                          service_name,
-                          )
-
-    def _update_file(self,
-                     file_,
-                     index,
-                     service_path,
-                     service_name,
-                     ):
-        if not hasattr(file_, 'file_type') or file_.file_type == "UnicodeOption":
-            if not hasattr(file_, 'source'):
-                file_.source = basename(file_.name)
-        elif not hasattr(file_, 'source'):
-            raise DictConsistencyError(_('attribute source mandatory for file with variable name '
-                                               'for {}').format(file_.name))
-
-
-class VariableAnnotator:
-    def __init__(self,
-                 objectspace,
-                 ):
-        self.objectspace = objectspace
-        self.convert_variable()
-        self.convert_helps()
-        self.convert_auto_freeze()
-        self.convert_separators()
-
-    def convert_variable(self):
-        def _convert_variable(variable,
-                              variable_type,
-                              ):
-            if not hasattr(variable, 'type'):
-                variable.type = 'string'
-            if variable.type != 'symlink' and not hasattr(variable, 'description'):
-                variable.description = variable.name
-            if hasattr(variable, 'value'):
-                for value in variable.value:
-                    if not hasattr(value, 'type'):
-                        value.type = variable.type
-                    value.name = CONVERT_OPTION.get(value.type, {}).get('func', str)(value.name)
-            for key, value in RENAME_ATTIBUTES.items():
-                setattr(variable, value, getattr(variable, key))
-                setattr(variable, key, None)
-            if variable_type == 'follower':
-                if variable.multi is True:
-                    variable.multi = 'submulti'
-                else:
-                    variable.multi = True
-
-        def _convert_valid_enum(namespace,
-                                variable,
-                                path,
-                                ):
-            if variable.type in FORCE_CHOICE:
-                check = self.objectspace.check(variable.xmlfiles)
-                check.name = 'valid_enum'
-                check.target = path
-                check.namespace = namespace
-                check.param = []
-                for value in FORCE_CHOICE[variable.type]:
-                    param = self.objectspace.param(variable.xmlfiles)
-                    param.text = value
-                    check.param.append(param)
-                if not hasattr(self.objectspace.space, 'constraints'):
-                    self.objectspace.space.constraints = self.objectspace.constraints(variable.xmlfiles)
-                    self.objectspace.space.constraints.namespace = namespace
-                if not hasattr(self.objectspace.space.constraints, 'check'):
-                    self.objectspace.space.constraints.check = []
-                self.objectspace.space.constraints.check.append(check)
-                variable.type = 'string'
-
-        def _valid_type(variable):
-            if variable.type not in CONVERT_OPTION:
-                xmlfiles = self.objectspace.display_xmlfiles(variable.xmlfiles)
-                raise DictConsistencyError(_(f'unvalid type "{variable.type}" for variable "{variable.name}" in {xmlfiles}'))
-
-        if not hasattr(self.objectspace.space, 'variables'):
-            return
-        for families in self.objectspace.space.variables.values():
-            namespace = families.name
-            if hasattr(families, 'family'):
-                families.doc = families.name
-                for family in families.family.values():
-                    family.doc = family.name
-                    for key, value in RENAME_ATTIBUTES.items():
-                        if hasattr(family, key):
-                            setattr(family, value, getattr(family, key))
-                            setattr(family, key, None)
-                    family.name = normalize_family(family.name)
-                    if hasattr(family, 'variable'):
-                        for variable in family.variable.values():
-                            if isinstance(variable, self.objectspace.Leadership):
-                                for idx, follower in enumerate(variable.variable):
-                                    if idx == 0:
-                                        variable_type = 'master'
-                                    else:
-                                        variable_type = 'follower'
-                                    path = '{}.{}.{}.{}'.format(namespace, normalize_family(family.name), variable.name, follower.name)
-                                    _convert_variable(follower,
-                                                      variable_type,
-                                                      )
-                                    _convert_valid_enum(namespace,
-                                                        follower,
-                                                        path,
-                                                        )
-                                    _valid_type(follower)
-                            else:
-                                path = '{}.{}.{}'.format(namespace, normalize_family(family.name), variable.name)
-                                _convert_variable(variable,
-                                                  'variable',
-                                                  )
-                                _convert_valid_enum(namespace,
-                                                    variable,
-                                                    path,
-                                                    )
-                                _valid_type(variable)
-
-    def convert_helps(self):
-        if not hasattr(self.objectspace.space, 'help'):
-            return
-        helps = self.objectspace.space.help
-        if hasattr(helps, 'variable'):
-            for hlp in helps.variable.values():
-                variable = self.objectspace.paths.get_variable_obj(hlp.name)
-                variable.help = hlp.text
-        if hasattr(helps, 'family'):
-            for hlp in helps.family.values():
-                variable = self.objectspace.paths.get_family_obj(hlp.name)
-                variable.help = hlp.text
-        del self.objectspace.space.help
-
-    def convert_auto_freeze(self):  # pylint: disable=C0111
-        def _convert_auto_freeze(variable, namespace):
-            if variable.auto_freeze:
-                new_condition = self.objectspace.condition(variable.xmlfiles)
-                new_condition.name = 'auto_hidden_if_not_in'
-                new_condition.namespace = namespace
-                new_condition.source = FREEZE_AUTOFREEZE_VARIABLE
-                new_param = self.objectspace.param(variable.xmlfiles)
-                new_param.text = 'oui'
-                new_condition.param = [new_param]
-                new_target = self.objectspace.target(variable.xmlfiles)
-                new_target.type = 'variable'
-                path = variable.namespace + '.' + normalize_family(family.name) + '.' + variable.name
-                new_target.name = path
-                new_condition.target = [new_target]
-                if not hasattr(self.objectspace.space.constraints, 'condition'):
-                    self.objectspace.space.constraints.condition = []
-                self.objectspace.space.constraints.condition.append(new_condition)
-        if hasattr(self.objectspace.space, 'variables'):
-            for variables in self.objectspace.space.variables.values():
-                if hasattr(variables, 'family'):
-                    namespace = variables.name
-                    for family in variables.family.values():
-                        if hasattr(family, 'variable'):
-                            for variable in family.variable.values():
-                                if isinstance(variable, self.objectspace.Leadership):
-                                    for follower in variable.variable:
-                                        _convert_auto_freeze(follower, namespace)
-                                else:
-                                    _convert_auto_freeze(variable, namespace)
-
-    def convert_separators(self):  # pylint: disable=C0111,R0201
-        if not hasattr(self.objectspace.space, 'variables'):
-            return
-        for family in self.objectspace.space.variables.values():
-            if not hasattr(family, 'separators'):
-                continue
-            if hasattr(family.separators, 'separator'):
-                for idx, separator in enumerate(family.separators.separator):
-                    option = self.objectspace.paths.get_variable_obj(separator.name)
-                    if hasattr(option, 'separator'):
-                        subpath = self.objectspace.paths.get_variable_path(separator.name,
-                                                                           separator.namespace,
-                                                                           )
-                        xmlfiles = self.objectspace.display_xmlfiles(separator.xmlfiles)
-                        raise DictConsistencyError(_(f'{subpath} already has a separator in {xmlfiles}'))
-                    option.separator = separator.text
-            del family.separators
-
-
-class ConstraintAnnotator:
-    def __init__(self,
-                 objectspace,
-                 eosfunc_file,
-                 ):
-        if not hasattr(objectspace.space, 'constraints'):
-            return
-        self.objectspace = objectspace
-        eosfunc = imp.load_source('eosfunc', eosfunc_file)
-        self.functions = dir(eosfunc)
-        self.functions.extend(INTERNAL_FUNCTIONS)
-        self.valid_enums = {}
-        if hasattr(self.objectspace.space.constraints, 'check'):
-            self.check_check()
-            self.check_replace_text()
-            self.check_valid_enum()
-            self.check_change_warning()
-            self.convert_check()
-        if hasattr(self.objectspace.space.constraints, 'condition'):
-            self.check_params_target()
-            self.filter_targets()
-            self.convert_xxxlist_to_variable()
-            self.check_condition_fallback_optional()
-            self.check_choice_option_condition()
-            self.remove_condition_with_empty_target()
-            self.convert_condition()
-        if hasattr(self.objectspace.space.constraints, 'fill'):
-            self.convert_fill()
-        self.remove_constraints()
-
-    def check_check(self):
-        remove_indexes = []
-        for check_idx, check in enumerate(self.objectspace.space.constraints.check):
-            if not check.name in self.functions:
-                xmlfiles = self.objectspace.display_xmlfiles(check.xmlfiles)
-                raise DictConsistencyError(_(f'cannot find check function "{check.name}" in {xmlfiles}'))
-            if hasattr(check, 'param'):
-                param_option_indexes = []
-                for idx, param in enumerate(check.param):
-                    if param.type == 'variable' and not self.objectspace.paths.path_is_defined(param.text):
-                        if param.optional is True:
-                            param_option_indexes.append(idx)
-                        else:
-                            xmlfiles = self.objectspace.display_xmlfiles(check.xmlfiles)
-                            raise DictConsistencyError(_(f'cannot find check param "{param.text}" in {xmlfiles}'))
-                    if param.type != 'variable':
-                        param.notraisepropertyerror = None
-                param_option_indexes = list(set(param_option_indexes))
-                param_option_indexes.sort(reverse=True)
-                for idx in param_option_indexes:
-                    check.param.pop(idx)
-                if check.param == []:
-                    remove_indexes.append(check_idx)
-        remove_indexes.sort(reverse=True)
-        for idx in remove_indexes:
-            del self.objectspace.space.constraints.check[idx]
-
-    def check_replace_text(self):
-        for check_idx, check in enumerate(self.objectspace.space.constraints.check):
-            namespace = check.namespace
-            if hasattr(check, 'param'):
-                for idx, param in enumerate(check.param):
-                    if param.type == 'variable':
-                        param.text = self.objectspace.paths.get_variable_path(param.text, namespace)
-            check.is_in_leadership = self.objectspace.paths.get_leader(check.target) != None
-            # let's replace the target by the path
-            check.target = self.objectspace.paths.get_variable_path(check.target, namespace)
-
-    def check_valid_enum(self):
-        remove_indexes = []
-        for idx, check in enumerate(self.objectspace.space.constraints.check):
-            if check.name == 'valid_enum':
-                if check.target in self.valid_enums:
-                    raise DictConsistencyError(_(f'valid_enum already set for {check.target}'))
-                if not hasattr(check, 'param'):
-                    raise DictConsistencyError(_(f'param is mandatory for a valid_enum of variable {check.target}'))
-                variable = self.objectspace.paths.get_variable_obj(check.target)
-                values = self.load_params_in_valid_enum(check.param,
-                                                        variable.name,
-                                                        variable.type,
-                                                        )
-                self._set_valid_enum(variable,
-                                     values,
-                                     variable.type,
-                                     check.target
-                                     )
-                remove_indexes.append(idx)
-        remove_indexes.sort(reverse=True)
-        for idx in remove_indexes:
-            del self.objectspace.space.constraints.check[idx]
-
-    def load_params_in_valid_enum(self,
-                                  params,
-                                  variable_name,
-                                  variable_type,
-                                  ):
-        has_variable = None
-        values = []
-        for param in params:
-            if param.type == 'variable':
-                if has_variable is not None:
-                    raise DictConsistencyError(_(f'only one "variable" parameter is allowed for valid_enum of variable {variable_name}'))
-                has_variable = True
-                variable = self.objectspace.paths.get_variable_obj(param.text)
-                if not variable.multi:
-                    raise DictConsistencyError(_(f'only multi "variable" parameter is allowed for valid_enum of variable {variable_name}'))
-                values = param.text
-            else:
-                if has_variable:
-                    raise DictConsistencyError(_(f'only one "variable" parameter is allowed for valid_enum of variable {variable_name}'))
-                if not hasattr(param, 'text'):
-                    if param.type == 'number':
-                        raise DictConsistencyError(_(f'value is mandatory for valid_enum of variable {variable_name}'))
-                    values.append(None)
-                else:
-                    values.append(param.text)
-        return values
-
-    def check_change_warning(self):
-        #convert level to "warnings_only"
-        for check in self.objectspace.space.constraints.check:
-            if check.level == 'warning':
-                check.warnings_only = True
-            else:
-                check.warnings_only = False
-            check.level = None
-
-    def _get_family_variables_from_target(self,
-                                          target,
-                                          ):
-        if target.type == 'variable':
-            variable = self.objectspace.paths.get_variable_obj(target.name)
-            family = self.objectspace.paths.get_family_obj(target.name.rsplit('.', 1)[0])
-            if isinstance(family, self.objectspace.Leadership) and family.name == variable.name:
-                return family, family.variable
-            return variable, [variable]
-        # it's a family
-        variable = self.objectspace.paths.get_family_obj(target.name)
-        return variable, list(variable.variable.values())
-
-    def check_params_target(self):
-        for condition in self.objectspace.space.constraints.condition:
-            if not hasattr(condition, 'target'):
-                raise DictConsistencyError(_('target is mandatory in condition'))
-            for target in condition.target:
-                if target.type.endswith('list') and condition.name not in ['disabled_if_in', 'disabled_if_not_in']:
-                    raise DictConsistencyError(_(f'target in condition for {target.type} not allow in {condition.name}'))
-
-    def filter_targets(self):  # pylint: disable=C0111
-        for condition_idx, condition in enumerate(self.objectspace.space.constraints.condition):
-            namespace = condition.namespace
-            for idx, target in enumerate(condition.target):
-                if target.type == 'variable':
-                    if condition.source == target.name:
-                        raise DictConsistencyError(_('target name and source name must be different: {}').format(condition.source))
-                    try:
-                        target.name = self.objectspace.paths.get_variable_path(target.name, namespace)
-                    except DictConsistencyError:
-                        # for optional variable
-                        pass
-                elif target.type == 'family':
-                    try:
-                        target.name = self.objectspace.paths.get_family_path(target.name, namespace)
-                    except KeyError:
-                        raise DictConsistencyError(_('cannot found family {}').format(target.name))
-
-    def convert_xxxlist_to_variable(self):  # pylint: disable=C0111
-        # transform *list to variable or family
-        for condition_idx, condition in enumerate(self.objectspace.space.constraints.condition):
-            new_targets = []
-            remove_targets = []
-            for target_idx, target in enumerate(condition.target):
-                if target.type.endswith('list'):
-                    listname = target.type
-                    listvars = self.objectspace.list_conditions.get(listname,
-                                                                    {}).get(target.name)
-                    if listvars:
-                        for listvar in listvars:
-                            variable = self.objectspace.paths.get_variable_obj(listvar)
-                            type_ = 'variable'
-                            new_target = self.objectspace.target(variable.xmlfiles)
-                            new_target.type = type_
-                            new_target.name = listvar
-                            new_target.index = target.index
-                            new_targets.append(new_target)
-                    remove_targets.append(target_idx)
-            remove_targets.sort(reverse=True)
-            for target_idx in remove_targets:
-                condition.target.pop(target_idx)
-            condition.target.extend(new_targets)
-
-    def check_condition_fallback_optional(self):
-        # a condition with a fallback **and** the source variable doesn't exist
-        remove_conditions = []
-        for idx, condition in enumerate(self.objectspace.space.constraints.condition):
-            # fallback
-            if condition.fallback is True and not self.objectspace.paths.path_is_defined(condition.source):
-                apply_action = False
-                if condition.name in ['disabled_if_in', 'mandatory_if_in', 'hidden_if_in']:
-                    apply_action = not condition.force_condition_on_fallback
-                else:
-                    apply_action = condition.force_inverse_condition_on_fallback
-                if apply_action:
-                    actions = self._get_condition_actions(condition.name)
-                    for target in condition.target:
-                        leader_or_variable, variables = self._get_family_variables_from_target(target)
-                        for action_idx, action in enumerate(actions):
-                            if action_idx == 0:
-                                setattr(leader_or_variable, action, True)
-                            else:
-                                for variable in variables:
-                                    setattr(variable, action, True)
-                remove_conditions.append(idx)
-                continue
-
-            remove_targets = []
-            # optional
-            for idx, target in enumerate(condition.target):
-                if target.optional is True and not self.objectspace.paths.path_is_defined(target.name):
-                    remove_targets.append(idx)
-            remove_targets = list(set(remove_targets))
-            remove_targets.sort(reverse=True)
-            for idx in remove_targets:
-                condition.target.pop(idx)
-        remove_conditions = list(set(remove_conditions))
-        remove_conditions.sort(reverse=True)
-        for idx in remove_conditions:
-            self.objectspace.space.constraints.condition.pop(idx)
-
-    def _get_condition_actions(self, condition_name):
-        if condition_name.startswith('disabled_if_'):
-            return ['disabled']
-        elif condition_name.startswith('hidden_if_'):
-            return ['hidden', 'frozen', 'force_default_on_freeze']
-        elif condition_name.startswith('mandatory_if_'):
-            return ['mandatory']
-        elif condition_name == 'auto_hidden_if_not_in':
-            return ['auto_frozen']
-
-    def check_choice_option_condition(self):
-        # remove condition for ChoiceOption that don't have param
-        remove_conditions = []
-        for condition_idx, condition in enumerate(self.objectspace.space.constraints.condition):
-            namespace = condition.namespace
-            condition.source = self.objectspace.paths.get_variable_path(condition.source, namespace, allow_source=True)
-            src_variable = self.objectspace.paths.get_variable_obj(condition.source)
-            valid_enum = None
-            if condition.source in self.valid_enums and self.valid_enums[condition.source]['type'] == 'string':
-                valid_enum = self.valid_enums[condition.source]['values']
-            if valid_enum is not None:
-                remove_param = []
-                for param_idx, param in enumerate(condition.param):
-                    if param.text not in valid_enum:
-                        remove_param.append(param_idx)
-                remove_param.sort(reverse=True)
-                for idx in remove_param:
-                    del condition.param[idx]
-                if condition.param == []:
-                    remove_targets = []
-                    for target in condition.target:
-                        leader_or_variable, variables = self._get_family_variables_from_target(target)
-                        if condition.name == 'disabled_if_not_in':
-                            leader_or_variable.disabled = True
-                        elif condition.name == 'hidden_if_not_in':
-                            leader_or_variable.hidden = True
-                            for variable in variables:
-                                variable.frozen = True
-                                variable.force_default_on_freeze = True
-                        elif condition.name == 'mandatory_if_not_in':
-                            variable.mandatory = True
-                    remove_targets = list(set(remove_targets))
-                    remove_targets.sort(reverse=True)
-                    for target_idx in remove_targets:
-                        condition.target.pop(target_idx)
-                    remove_conditions.append(condition_idx)
-        remove_conditions = list(set(remove_conditions))
-        remove_conditions.sort(reverse=True)
-        for idx in remove_conditions:
-            self.objectspace.space.constraints.condition.pop(idx)
-
-    def remove_condition_with_empty_target(self):
-        remove_conditions = []
-        for condition_idx, condition in enumerate(self.objectspace.space.constraints.condition):
-            if not condition.target:
-                remove_conditions.append(condition_idx)
-        remove_conditions = list(set(remove_conditions))
-        remove_conditions.sort(reverse=True)
-        for idx in remove_conditions:
-            self.objectspace.space.constraints.condition.pop(idx)
-
-    def convert_condition(self):
-        for condition in self.objectspace.space.constraints.condition:
-            inverse = condition.name.endswith('_if_not_in')
-            actions = self._get_condition_actions(condition.name)
-            for param in condition.param:
-                if hasattr(param, 'text'):
-                    param = param.text
-                else:
-                    param = None
-                for target in condition.target:
-                    leader_or_variable, variables = self._get_family_variables_from_target(target)
-                    # if option is already disable, do not apply disable_if_in
-                    if hasattr(leader_or_variable, actions[0]) and getattr(leader_or_variable, actions[0]) is True:
-                        continue
-                    for idx, action in enumerate(actions):
-                        prop = self.objectspace.property_(leader_or_variable.xmlfiles)
-                        prop.type = 'calculation'
-                        prop.inverse = inverse
-                        prop.source = condition.source
-                        prop.expected = param
-                        prop.name = action
-                        if idx == 0:
-                            if not hasattr(leader_or_variable, 'property'):
-                                leader_or_variable.property = []
-                            leader_or_variable.property.append(prop)
-                        else:
-                            for variable in variables:
-                                if not hasattr(variable, 'property'):
-                                    variable.property = []
-                                variable.property.append(prop)
-        del self.objectspace.space.constraints.condition
-
-    def _set_valid_enum(self, variable, values, type_, target):
-        # value for choice's variable is mandatory
-        variable.mandatory = True
-        # build choice
-        variable.choice = []
-        if isinstance(values, str):
-            choice = self.objectspace.choice()
-            choice.type = 'calculation'
-            choice.name = values
-            variable.choice.append(choice)
-        else:
-            self.valid_enums[target] = {'type': type_,
-                                        'values': values,
-                                        }
-            choices = []
-            for value in values:
-                choice = self.objectspace.choice(variable.xmlfiles)
-                try:
-                    if value is not None:
-                        choice.name = CONVERT_OPTION[type_].get('func', str)(value)
-                    else:
-                        choice.name = value
-                except:
-                    raise DictConsistencyError(_(f'unable to change type of a valid_enum entry "{value}" is not a valid "{type_}" for "{variable.name}"'))
-                if choice.name == '':
-                    choice.name = None
-                choices.append(choice.name)
-                choice.type = type_
-                variable.choice.append(choice)
-            # check value or set first choice value has default value
-            if hasattr(variable, 'value'):
-                for value in variable.value:
-                    value.type = type_
-                    try:
-                        cvalue = CONVERT_OPTION[type_].get('func', str)(value.name)
-                    except:
-                        raise DictConsistencyError(_(f'unable to change type of value "{value}" is not a valid "{type_}" for "{variable.name}"'))
-                    if cvalue not in choices:
-                        raise DictConsistencyError(_('value "{}" of variable "{}" is not in list of all expected values ({})').format(value.name, variable.name, choices))
-            else:
-                new_value = self.objectspace.value(variable.xmlfiles)
-                new_value.name = choices[0]
-                new_value.type = type_
-                variable.value = [new_value]
-        if not variable.choice:
-            raise DictConsistencyError(_('empty valid enum is not allowed for variable {}').format(variable.name))
-        variable.type = 'choice'
-
-    def convert_check(self):
-        for check in self.objectspace.space.constraints.check:
-            variable = self.objectspace.paths.get_variable_obj(check.target)
-            name = check.name
-            if name == 'valid_entier':
-                if not hasattr(check, 'param'):
-                    raise DictConsistencyError(_('{} must have, at least, 1 param').format(name))
-                for param in check.param:
-                    if param.type not in ['string', 'number']:
-                        raise DictConsistencyError(_(f'param in "valid_entier" must not be a "{param.type}"'))
-                    if param.name == 'mini':
-                        variable.min_number = int(param.text)
-                    elif param.name == 'maxi':
-                        variable.max_number = int(param.text)
-                    else:
-                        raise DictConsistencyError(_(f'unknown parameter {param.text} in check "valid_entier" for variable {check.target}'))
-            else:
-                check_ = self.objectspace.check(variable.xmlfiles)
-                if name == 'valid_differ':
-                    name = 'valid_not_equal'
-                elif name == 'valid_network_netmask':
-                    params_len = 1
-                    if len(check.param) != params_len:
-                        xmlfiles = self.objectspace.display_xmlfiles(check.xmlfiles)
-                        raise DictConsistencyError(_(f'{name} must have {params_len} param in {xmlfiles}'))
-                elif name == 'valid_ipnetmask':
-                    params_len = 1
-                    if len(check.param) != params_len:
-                        xmlfiles = self.objectspace.display_xmlfiles(check.xmlfiles)
-                        raise DictConsistencyError(_(f'{name} must have {params_len} param in {xmlfiles}'))
-                    name = 'valid_ip_netmask'
-                elif name == 'valid_broadcast':
-                    params_len = 2
-                    if len(check.param) != params_len:
-                        xmlfiles = self.objectspace.display_xmlfiles(check.xmlfiles)
-                        raise DictConsistencyError(_(f'{name} must have {params_len} param in {xmlfiles}'))
-                elif name == 'valid_in_network':
-                    if len(check.param) not in (1, 2):
-                        params_len = 2
-                        xmlfiles = self.objectspace.display_xmlfiles(check.xmlfiles)
-                        raise DictConsistencyError(_(f'{name} must have {params_len} param in {xmlfiles}'))
-                check_.name = name
-                check_.warnings_only = check.warnings_only
-                if hasattr(check, 'param'):
-                    check_.param = check.param
-                if not hasattr(variable, 'check'):
-                    variable.check = []
-                variable.check.append(check_)
-        del self.objectspace.space.constraints.check
-
-    def convert_fill(self):  # pylint: disable=C0111,R0912
-        # sort fill/auto by index
-        fills = {fill.index: fill for idx, fill in enumerate(self.objectspace.space.constraints.fill)}
-        indexes = list(fills.keys())
-        indexes.sort()
-        targets = []
-        for idx in indexes:
-            fill = fills[idx]
-            # test if it's redefined calculation
-            if fill.target in targets and not fill.redefine:
-                raise DictConsistencyError(_(f"A fill already exists for the target: {fill.target}"))
-            targets.append(fill.target)
-            #
-            if fill.name not in self.functions:
-                raise DictConsistencyError(_('cannot find fill function {}').format(fill.name))
-
-            namespace = fill.namespace
-            # let's replace the target by the path
-            fill.target, suffix = self.objectspace.paths.get_variable_path(fill.target,
-                                                                           namespace,
-                                                                           with_suffix=True,
-                                                                           )
-            if suffix is not None:
-                raise DictConsistencyError(_(f'Cannot add fill function to "{fill.target}" only with the suffix "{suffix}"'))
-            variable = self.objectspace.paths.get_variable_obj(fill.target)
-            value = self.objectspace.value(variable.xmlfiles)
-            value.type = 'calculation'
-            value.name = fill.name
-            if hasattr(fill, 'param'):
-                param_to_delete = []
-                for fill_idx, param in enumerate(fill.param):
-                    if param.type not in ['suffix', 'string'] and not hasattr(param, 'text'):
-                        raise DictConsistencyError(_(f"All '{param.type}' variables must have a value in order to calculate {fill.target}"))
-                    if param.type == 'suffix' and hasattr(param, 'text'):
-                        raise DictConsistencyError(_(f"All '{param.type}' variables must not have a value in order to calculate {fill.target}"))
-                    if param.type == 'string':
-                        if not hasattr(param, 'text'):
-                            param.text = None
-                    if param.type == 'variable':
-                        try:
-                            param.text, suffix = self.objectspace.paths.get_variable_path(param.text,
-                                                                                          namespace,
-                                                                                          with_suffix=True,
-                                                                                          )
-                            if suffix:
-                                param.suffix = suffix
-                        except DictConsistencyError as err:
-                            if param.optional is False:
-                                raise err
-                            param_to_delete.append(fill_idx)
-                            continue
-                    else:
-                        param.notraisepropertyerror = None
-                param_to_delete.sort(reverse=True)
-                for param_idx in param_to_delete:
-                    fill.param.pop(param_idx)
-                value.param = fill.param
-            variable.value = [value]
-        del self.objectspace.space.constraints.fill
-
-    def remove_constraints(self):
-        if hasattr(self.objectspace.space.constraints, 'index'):
-            del self.objectspace.space.constraints.index
-        del self.objectspace.space.constraints.namespace
-        del self.objectspace.space.constraints.xmlfiles
-        if vars(self.objectspace.space.constraints):
-            raise Exception('constraints again?')
-        del self.objectspace.space.constraints
-
-
-class FamilyAnnotator:
-    def __init__(self,
-                 objectspace,
-                 ):
-        self.objectspace = objectspace
-        self.remove_empty_families()
-        self.change_variable_mode()
-        self.change_family_mode()
-        self.dynamic_families()
-
-    def remove_empty_families(self):  # pylint: disable=C0111,R0201
-        if hasattr(self.objectspace.space, 'variables'):
-            for family in self.objectspace.space.variables.values():
-                if hasattr(family, 'family'):
-                    space = family.family
-                    removed_families = []
-                    for family_name, family in space.items():
-                        if not hasattr(family, 'variable') or len(family.variable) == 0:
-                            removed_families.append(family_name)
-                    for family_name in removed_families:
-                        del space[family_name]
-
-    def change_family_mode(self):  # pylint: disable=C0111
-        if not hasattr(self.objectspace.space, 'variables'):
-            return
-        for family in self.objectspace.space.variables.values():
-            if hasattr(family, 'family'):
-                for family in family.family.values():
-                    mode = modes_level[-1]
-                    for variable in family.variable.values():
-                        if isinstance(variable, self.objectspace.Leadership):
-                            variable_mode = variable.variable[0].mode
-                            variable.variable[0].mode = None
-                            variable.mode = variable_mode
-                        else:
-                            variable_mode = variable.mode
-                        if variable_mode is not None and modes[mode] > modes[variable_mode]:
-                            mode = variable_mode
-                    family.mode = mode
-
-    def dynamic_families(self):  # pylint: disable=C0111
-        if not hasattr(self.objectspace.space, 'variables'):
-            return
-        for family in self.objectspace.space.variables.values():
-            if hasattr(family, 'family'):
-                for family in family.family.values():
-                    if 'dynamic' in vars(family):
-                        namespace = self.objectspace.paths.get_variable_namespace(family.dynamic)
-                        varpath = self.objectspace.paths.get_variable_path(family.dynamic, namespace)
-                        family.dynamic = varpath
-
-    def annotate_variable(self, variable, family_mode, path, is_follower=False):
-        # if the variable is mandatory and doesn't have any value
-        # then the variable's mode is set to 'basic'
-        if not hasattr(variable, 'value') and variable.type == 'boolean':
-            new_value = self.objectspace.value(variable.xmlfiles)
-            new_value.name = True
-            new_value.type = 'boolean'
-            variable.value = [new_value]
-        if hasattr(variable, 'value') and variable.value:
-            has_value = True
-            for value in variable.value:
-                if value.type == 'calculation':
-                    has_value = False
-                    has_variable = False
-                    if hasattr(value, 'param'):
-                        for param in value.param:
-                            if param.type == 'variable':
-                                has_variable = True
-                                break
-                    #if not has_variable:
-                    #    # if one parameter is a variable, let variable choice if it's mandatory
-                    #    variable.mandatory = True
-            if has_value:
-                # if has value but without any calculation
-                variable.mandatory = True
-        if variable.mandatory is True and (not hasattr(variable, 'value') or is_follower):
-            variable.mode = modes_level[0]
-        if variable.mode != None and modes[variable.mode] < modes[family_mode] and (not is_follower or variable.mode != modes_level[0]):
-            variable.mode = family_mode
-        if variable.hidden is True:
-            variable.frozen = True
-            if not variable.auto_save is True and 'force_default_on_freeze' not in vars(variable):
-                variable.force_default_on_freeze = True
-
-    def change_variable_mode(self):  # pylint: disable=C0111
-        if not hasattr(self.objectspace.space, 'variables'):
-            return
-        for variables in self.objectspace.space.variables.values():
-            namespace = variables.name
-            if hasattr(variables, 'family'):
-                for family in variables.family.values():
-                    family_mode = family.mode
-                    if hasattr(family, 'variable'):
-                        for variable in family.variable.values():
-
-                            if isinstance(variable, self.objectspace.Leadership):
-                                mode = modes_level[-1]
-                                for idx, follower in enumerate(variable.variable):
-                                    if follower.auto_save is True:
-                                        raise DictConsistencyError(_(f'leader/followers {follower.name} could not be auto_save'))
-                                    if follower.auto_freeze is True:
-                                        raise DictConsistencyError(_('leader/followers {follower.name} could not be auto_freeze'))
-                                    is_follower = idx != 0
-                                    path = '{}.{}.{}'.format(family.path, variable.name, follower.name)
-                                    self.annotate_variable(follower, family_mode, path, is_follower)
-                                    # leader's mode is minimum level
-                                    if modes[variable.variable[0].mode] > modes[follower.mode]:
-                                        follower.mode = variable.variable[0].mode
-                                variable.mode = variable.variable[0].mode
-                            else:
-                                # auto_save's variable is set in 'basic' mode if its mode is 'normal'
-                                if variable.auto_save is True and variable.mode != modes_level[-1]:
-                                    variable.mode = modes_level[0]
-                                # auto_freeze's variable is set in 'basic' mode if its mode is 'normal'
-                                if variable.auto_freeze is True and variable.mode != modes_level[-1]:
-                                    variable.mode = modes_level[0]
-                                path = '{}.{}'.format(family.path, variable.name)
-                                self.annotate_variable(variable, family_mode, path)
-
-class PropertyAnnotator:
-    def __init__(self, objectspace):
-        self.objectspace = objectspace
-        self.convert_annotator()
-
-    def convert_property(self,
-                         variable,
-                         ):
-        properties = []
-        for prop in PROPERTIES:
-            if hasattr(variable, prop):
-                if getattr(variable, prop) == True:
-                    for subprop in CONVERT_PROPERTIES.get(prop, [prop]):
-                        properties.append(subprop)
-                setattr(variable, prop, None)
-        if hasattr(variable, 'mode') and variable.mode:
-            properties.append(variable.mode)
-            variable.mode = None
-        if properties:
-            variable.properties = frozenset(properties)
-
-    def convert_annotator(self):  # pylint: disable=C0111
-        if hasattr(self.objectspace.space, 'services'):
-            self.convert_property(self.objectspace.space.services)
-            for services in self.objectspace.space.services.service.values():
-                self.convert_property(services)
-                for service in vars(services).values():
-                    if isinstance(service, self.objectspace.family):
-                        self.convert_property(service)
-                        if hasattr(service, 'family'):
-                            self.convert_property(service)
-                            for family in service.family:
-                                self.convert_property(family)
-                                if hasattr(family, 'variable'):
-                                    for variable in family.variable:
-                                        self.convert_property(variable)
-        if hasattr(self.objectspace.space, 'variables'):
-            for variables in self.objectspace.space.variables.values():
-                if hasattr(variables, 'family'):
-                    for family in variables.family.values():
-                        self.convert_property(family)
-                        if hasattr(family, 'variable'):
-                            for variable in family.variable.values():
-                                if isinstance(variable, self.objectspace.Leadership):
-                                    self.convert_property(variable)
-                                    for follower in variable.variable:
-                                        self.convert_property(follower)
-                                else:
-                                    self.convert_property(variable)

src/rougail/annotator/__init__.py

@@ -0,0 +1,57 @@
+"""Annotate dictionaries
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from .group import GroupAnnotator
+from .service import ServiceAnnotator
+from .variable import VariableAnnotator, CONVERT_OPTION
+from .check import CheckAnnotator
+from .value import ValueAnnotator
+from .condition import ConditionAnnotator
+from .fill import FillAnnotator
+from .family import FamilyAnnotator
+from .property import PropertyAnnotator
+
+class SpaceAnnotator:  # pylint: disable=R0903
+    """Transformations applied on a object instance
+    """
+    def __init__(self, objectspace, eosfunc_file):
+        self.objectspace = objectspace
+        GroupAnnotator(objectspace)
+        ServiceAnnotator(objectspace)
+        VariableAnnotator(objectspace)
+        CheckAnnotator(objectspace,
+                       eosfunc_file,
+                       )
+        ConditionAnnotator(objectspace)
+        FillAnnotator(objectspace,
+                      eosfunc_file,
+                      )
+        ValueAnnotator(objectspace)
+        FamilyAnnotator(objectspace)
+        PropertyAnnotator(objectspace)
+
+
+__all__ = ('SpaceAnnotator', 'CONVERT_OPTION')

src/rougail/annotator/check.py

@@ -0,0 +1,242 @@
+"""Annotate check
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from typing import List, Any
+
+from .target import TargetAnnotator
+from .param import ParamAnnotator
+
+from ..utils import load_modules
+from ..i18n import _
+from ..error import DictConsistencyError, display_xmlfiles
+
+INTERNAL_FUNCTIONS = ['valid_enum', 'valid_in_network', 'valid_differ', 'valid_entier']
+
+class CheckAnnotator(TargetAnnotator, ParamAnnotator):
+    """Annotate check
+    """
+    def __init__(self,
+                 objectspace,
+                 eosfunc_file,
+                 ):
+        if not hasattr(objectspace.space, 'constraints') or \
+                not hasattr(objectspace.space.constraints, 'check'):
+            return
+        self.objectspace = objectspace
+        self.let_none = True
+        self.only_variable = True
+        self.functions = dir(load_modules(eosfunc_file))
+        self.functions.extend(INTERNAL_FUNCTIONS)
+        self.functions.extend(self.objectspace.rougailconfig['internal_functions'])
+        self.target_is_uniq = False
+        self.allow_function = True
+        self.convert_target(self.objectspace.space.constraints.check)
+        self.convert_param(self.objectspace.space.constraints.check)
+        self.check_check()
+        self.check_valid_enum()
+        self.check_change_warning()
+        self.convert_valid_entier()
+        self.convert_check()
+        del objectspace.space.constraints.check
+
+    def valid_type_validation(self,
+                              obj,
+                              ) -> None:
+        variable_type = None
+        if obj.name == 'valid_enum':
+            for target in obj.target:
+                variable_type = target.name.type
+        return variable_type
+
+    def check_check(self):  # pylint: disable=R0912
+        """valid and manage <check>
+        """
+        remove_indexes = []
+        for check_idx, check in enumerate(self.objectspace.space.constraints.check):
+            if not check.name in self.functions:
+                msg = _(f'cannot find check function "{check.name}"')
+                raise DictConsistencyError(msg, 1, check.xmlfiles)
+            if hasattr(check, 'param') and check.param == []:
+                remove_indexes.append(check_idx)
+        remove_indexes.sort(reverse=True)
+        for idx in remove_indexes:
+            del self.objectspace.space.constraints.check[idx]
+
+    def check_valid_enum(self):
+        """verify valid_enum
+        """
+        remove_indexes = []
+        for idx, check in enumerate(self.objectspace.space.constraints.check):
+            if check.name != 'valid_enum':
+                continue
+            for target in check.target:
+                if target.name.path in self.objectspace.valid_enums:
+                    check_xmlfiles = display_xmlfiles(self.objectspace.valid_enums\
+                            [target.name.path]['xmlfiles'])
+                    msg = _(f'valid_enum already set in {check_xmlfiles} '
+                            f'for "{target.name.name}", you may have forget remove_check')
+                    raise DictConsistencyError(msg, 3, check.xmlfiles)
+                if not hasattr(check, 'param'):
+                    msg = _(f'param is mandatory for a valid_enum of variable "{target.name.name}"')
+                    raise DictConsistencyError(msg, 4, check.xmlfiles)
+                variable_type = target.name.type
+                values = self._set_valid_enum(target.name,
+                                              check,
+                                              )
+                if values:
+                    if hasattr(target.name, 'value'):
+                        # check value
+                        self.check_valid_enum_value(target.name, values)
+                    else:
+                        # no value, set the first choice as default value
+                        new_value = self.objectspace.value(check.xmlfiles)
+                        new_value.name = values[0]
+                        new_value.type = variable_type
+                        target.name.value = [new_value]
+            remove_indexes.append(idx)
+        remove_indexes.sort(reverse=True)
+        for idx in remove_indexes:
+            del self.objectspace.space.constraints.check[idx]
+
+    def _set_valid_enum(self,
+                        variable,
+                        check,
+                        ) -> List[Any]:
+        # build choice
+        variable.values = []
+        variable.ori_type = variable.type
+        variable.type = 'choice'
+
+        has_variable = False
+        values = []
+        has_nil = False
+        is_function = False
+        for param in check.param:
+            if has_variable:
+                msg = _(f'only one "variable" parameter is allowed for valid_enum '
+                        f'of variable "{variable.name}"')
+                raise DictConsistencyError(msg, 5, param.xmlfiles)
+            if param.type == 'function':
+                is_function = True
+                choice = self.objectspace.choice(variable.xmlfiles)
+                choice.name = param.text
+                choice.type = 'function'
+                choice.param = []
+                variable.values.append(choice)
+                continue
+            if is_function:
+                variable.values[0].param.append(param)
+                continue
+            param_type = variable.ori_type
+            if param.type == 'variable':
+                has_variable = True
+                if param.optional is True:
+                    msg = _(f'optional parameter in valid_enum for variable "{variable.name}" '
+                            f'is not allowed')
+                    raise DictConsistencyError(msg, 14, param.xmlfiles)
+                if not param.text.multi:
+                    msg = _(f'only multi "variable" parameter is allowed for valid_enum '
+                            f'of variable "{variable.name}"')
+                    raise DictConsistencyError(msg, 6, param.xmlfiles)
+                param_type = 'variable'
+            elif param.type == 'nil':
+                has_nil = True
+            values.append(param.text)
+            choice = self.objectspace.choice(variable.xmlfiles)
+            choice.name = param.text
+            choice.type = param_type
+            variable.values.append(choice)
+        if is_function:
+            return None
+        if 'mandatory' not in vars(variable):
+            variable.mandatory = not has_nil
+        elif variable.mandatory is False:
+            choice = self.objectspace.choice(variable.xmlfiles)
+            choice.name = None
+            choice.type = 'nil'
+            variable.values.append(choice)
+        if has_variable:
+            return None
+        self.objectspace.valid_enums[variable.path] = {'type': variable.ori_type,
+                                                       'values': values,
+                                                       'xmlfiles': check.xmlfiles,
+                                                       }
+        return values
+
+    @staticmethod
+    def check_valid_enum_value(variable,
+                               values,
+                               ) -> None:
+        """check that values in valid_enum are valid
+        """
+        for value in variable.value:
+            if value.name not in values:
+                msg = _(f'value "{value.name}" of variable "{variable.name}" is not in list '
+                        f'of all expected values ({values})')
+                raise DictConsistencyError(msg, 15, value.xmlfiles)
+
+    def check_change_warning(self):
+        """convert level to "warnings_only"
+        """
+        for check in self.objectspace.space.constraints.check:
+            check.warnings_only = check.level == 'warning'
+            check.level = None
+
+    def convert_valid_entier(self) -> None:
+        """valid and manage <check>
+        """
+        remove_indexes = []
+        for check_idx, check in enumerate(self.objectspace.space.constraints.check):
+            if not check.name == 'valid_entier':
+                continue
+            remove_indexes.append(check_idx)
+            if not hasattr(check, 'param'):
+                msg = _(f'{check.name} must have, at least, 1 param')
+                raise DictConsistencyError(msg, 17, check.xmlfiles)
+            for param in check.param:
+                if param.type != 'number':
+                    msg = _(f'param in "valid_entier" must be an "integer", not "{param.type}"')
+                    raise DictConsistencyError(msg, 18, check.xmlfiles)
+                for target in check.target:
+                    if param.name == 'mini':
+                        target.name.min_number = int(param.text)
+                    elif param.name == 'maxi':
+                        target.name.max_number = int(param.text)
+                    else:
+                        msg = _(f'unknown parameter "{param.name}" in check "valid_entier"')
+                        raise DictConsistencyError(msg, 19, check.xmlfiles)
+        remove_indexes.sort(reverse=True)
+        for idx in remove_indexes:
+            del self.objectspace.space.constraints.check[idx]
+
+    def convert_check(self) -> None:
+        """valid and manage <check>
+        """
+        for check in self.objectspace.space.constraints.check:
+            for target in check.target:
+                if not hasattr(target.name, 'validators'):
+                    target.name.validators = []
+                target.name.validators.append(check)

src/rougail/annotator/condition.py

@@ -0,0 +1,369 @@
+"""Annotate condition
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from typing import List
+
+
+from ..i18n import _
+from ..error import DictConsistencyError
+
+from .target import TargetAnnotator
+from .param import ParamAnnotator
+from .variable import Walk
+
+
+class ConditionAnnotator(TargetAnnotator, ParamAnnotator, Walk):
+    """Annotate condition
+    """
+    def __init__(self,
+                 objectspace,
+                 ):
+        self.objectspace = objectspace
+        self.force_service_value = {}
+        if hasattr(objectspace.space, 'variables'):
+            self.convert_auto_freeze()
+        if not hasattr(objectspace.space, 'constraints') or \
+                not hasattr(self.objectspace.space.constraints, 'condition'):
+            return
+        self.target_is_uniq = False
+        self.only_variable = False
+        self.allow_function = False
+        self.convert_target(self.objectspace.space.constraints.condition)
+        self.check_condition_optional()
+        self.convert_condition_source()
+        self.convert_param(self.objectspace.space.constraints.condition)
+        self.check_source_target()
+        self.convert_xxxlist()
+        self.check_choice_option_condition()
+        self.remove_condition_with_empty_target()
+        self.convert_condition()
+
+    def valid_type_validation(self,
+                              obj,
+                              ) -> None:
+        if obj.source.type == 'choice':
+            return obj.source.ori_type
+        return obj.source.type
+
+    def convert_auto_freeze(self):
+        """convert auto_freeze
+        only if auto_freeze_variable is True this variable is frozen
+        """
+        for variable in self.get_variables():
+            if not variable.auto_freeze and not variable.auto_save:
+                continue
+            #if variable.namespace != self.objectspace.rougailconfig['variable_namespace']:
+            #    msg = _(f'auto_freeze is not allowed in extra "{variable.namespace}"')
+            #    raise DictConsistencyError(msg, 49, variable.xmlfiles)
+            variable.force_store_value = True
+            if variable.auto_save:
+                continue
+            new_condition = self.objectspace.condition(variable.xmlfiles)
+            new_condition.name = 'auto_frozen_if_in'
+            new_condition.namespace = variable.namespace
+            new_condition.source = self.objectspace.rougailconfig['auto_freeze_variable']
+            new_param = self.objectspace.param(variable.xmlfiles)
+            new_param.text = True
+            new_condition.param = [new_param]
+            new_target = self.objectspace.target(variable.xmlfiles)
+            new_target.type = 'variable'
+            new_target.name = variable.path
+            new_condition.target = [new_target]
+            if not hasattr(self.objectspace.space, 'constraints'):
+                self.objectspace.space.constraints = self.objectspace.constraints(variable.xmlfiles)
+            if not hasattr(self.objectspace.space.constraints, 'condition'):
+                self.objectspace.space.constraints.condition = []
+            self.objectspace.space.constraints.condition.append(new_condition)
+
+    def check_source_target(self):
+        """verify that source != target in condition
+        """
+        for condition in self.objectspace.space.constraints.condition:
+            for target in condition.target:
+                if target.type == 'variable' and \
+                        condition.source.path == target.name.path:
+                    msg = _('target name and source name must be different: '
+                            f'{condition.source.path}')
+                    raise DictConsistencyError(msg, 11, condition.xmlfiles)
+
+    def check_condition_optional(self):
+        """a condition with a optional **and** the source variable doesn't exist
+        """
+        remove_conditions = []
+        for idx, condition in enumerate(self.objectspace.space.constraints.condition):
+            if condition.optional is False or \
+                    self.objectspace.paths.path_is_defined(condition.source):
+                continue
+            remove_conditions.append(idx)
+            if (hasattr(condition, 'apply_on_fallback') and condition.apply_on_fallback) or \
+                    (not hasattr(condition, 'apply_on_fallback') and \
+                     condition.name.endswith('_if_in')):
+                self.force_actions_to_variable(condition)
+        remove_conditions.sort(reverse=True)
+        for idx in remove_conditions:
+            self.objectspace.space.constraints.condition.pop(idx)
+
+    def force_actions_to_variable(self,
+                                  condition: 'self.objectspace.condition',
+                                  ) -> None:
+        """force property to a variable
+        for example disabled_if_not_in => variable.disabled = True
+        """
+        actions = self.get_actions_from_condition(condition.name)
+        for target in condition.target:
+            main_action = actions[0]
+            if target.type.endswith('list'):
+                self.force_service_value[target.name] = main_action != 'disabled'
+                continue
+            leader_or_var, variables = self._get_family_variables_from_target(target)
+            setattr(leader_or_var, main_action, True)
+            for action in actions[1:]:
+                for variable in variables:
+                    setattr(variable, action, True)
+
+    @staticmethod
+    def get_actions_from_condition(condition_name: str) -> List[str]:
+        """get action's name from a condition
+        """
+        if condition_name.startswith('hidden_if_'):
+            return ['hidden', 'frozen', 'force_default_on_freeze']
+        if condition_name == 'auto_frozen_if_in':
+            return ['frozen']
+        return [condition_name.split('_', 1)[0]]
+
+    def _get_family_variables_from_target(self,
+                                          target,
+                                          ):
+        if target.type == 'variable':
+            if not self.objectspace.paths.is_leader(target.name.path):
+                return target.name, [target.name]
+            # it's a leader, so apply property to leadership
+            family_name = self.objectspace.paths.get_variable_family_path(target.name.path)
+            family = self.objectspace.paths.get_family(family_name,
+                                                       target.name.namespace,
+                                                       )
+            return family, family.variable
+        # it's a family
+        variable = self.objectspace.paths.get_family(target.name.path,
+                                                     target.namespace,
+                                                     )
+        if hasattr(variable, 'variable'):
+            return variable, list(variable.variable.values())
+        return variable, []
+
+    def convert_xxxlist(self):
+        """transform *list to variable or family
+        """
+        fills = {}
+        for condition in self.objectspace.space.constraints.condition:
+            remove_targets = []
+            for target_idx, target in enumerate(condition.target):
+                if target.type.endswith('list'):
+                    listvars = self.objectspace.list_conditions.get(target.type,
+                                                                    {}).get(target.name)
+                    if listvars:
+                        self._convert_xxxlist_to_fill(condition,
+                                                      target,
+                                                      listvars,
+                                                      fills,
+                                                      )
+                    elif not target.optional:
+                        msg = f'cannot found target "{target.type}" "{target.name}"'
+                        raise DictConsistencyError(_(msg), 2, target.xmlfiles)
+                    remove_targets.append(target_idx)
+            remove_targets.sort(reverse=True)
+            for target_idx in remove_targets:
+                condition.target.pop(target_idx)
+
+    def _convert_xxxlist_to_fill(self,
+                                 condition: 'self.objectspace.condition',
+                                 target: 'self.objectspace.target',
+                                 listvars: list,
+                                 fills: dict,
+                                 ):
+        for listvar in listvars:
+            if target.name in self.force_service_value:
+                listvar.default = self.force_service_value[target.name]
+                continue
+            if listvar.path in fills:
+                fill = fills[listvar.path]
+                or_needed = True
+                for param in fill.param:
+                    if hasattr(param, 'name') and \
+                            param.name == 'condition_operator':
+                        or_needed = False
+                        break
+                fill.index += 1
+            else:
+                fill = self.objectspace.fill(target.xmlfiles)
+                new_target = self.objectspace.target(target.xmlfiles)
+                new_target.name = listvar.path
+                fill.target = [new_target]
+                fill.name = 'calc_value'
+                fill.namespace = 'services'
+                fill.index = 0
+                if not hasattr(self.objectspace.space.constraints, 'fill'):
+                    self.objectspace.space.constraints.fill = []
+                self.objectspace.space.constraints.fill.append(fill)
+                fills[listvar.path] = fill
+                param1 = self.objectspace.param(target.xmlfiles)
+                param1.text = False
+                param1.type = 'boolean'
+                param2 = self.objectspace.param(target.xmlfiles)
+                param2.name = 'default'
+                param2.text = True
+                param2.type = 'boolean'
+                fill.param = [param1, param2]
+                or_needed = len(condition.param) != 1
+            if len(condition.param) == 1:
+                values = getattr(condition.param[0], 'text', None)
+                param_type = condition.param[0].type
+            else:
+                values = tuple([getattr(param, 'text', None) for param in condition.param])
+                param_type = None
+                for param in condition.param:
+                    if param_type is None or param_type == 'nil':
+                        param_type = param.type
+            param3 = self.objectspace.param(target.xmlfiles)
+            param3.name = f'condition_{fill.index}'
+            param3.type = 'variable'
+            param3.text = condition.source.path
+            param3.propertyerror = False
+            fill.param.append(param3)
+            param4 = self.objectspace.param(target.xmlfiles)
+            param4.name = f'expected_{fill.index}'
+            param4.text = values
+            param4.type = param_type
+            fill.param.append(param4)
+            if condition.name != 'disabled_if_in':
+                param5 = self.objectspace.param(target.xmlfiles)
+                param5.name = f'reverse_condition_{fill.index}'
+                param5.text = True
+                param5.type = 'boolean'
+                fill.param.append(param5)
+            if or_needed:
+                param6 = self.objectspace.param(target.xmlfiles)
+                param6.name = 'condition_operator'
+                param6.text = 'OR'
+                fill.param.append(param6)
+
+    def convert_condition_source(self):
+        """remove condition for ChoiceOption that don't have param
+        """
+        for condition in self.objectspace.space.constraints.condition:
+            try:
+                condition.source = self.objectspace.paths.get_variable(condition.source)
+            except DictConsistencyError as err:
+                if err.errno == 36:
+                    msg = _(f'the source "{condition.source}" in condition cannot be a dynamic '
+                            f'variable')
+                    raise DictConsistencyError(msg, 20, condition.xmlfiles) from err
+                if err.errno == 42:
+                    msg = _(f'the source "{condition.source}" in condition is an unknown variable')
+                    raise DictConsistencyError(msg, 23, condition.xmlfiles) from err
+                raise err from err  # pragma: no cover
+
+    def check_choice_option_condition(self):
+        """remove condition of ChoiceOption that doesn't match
+        """
+        remove_conditions = []
+        for condition_idx, condition in enumerate(self.objectspace.space.constraints.condition):
+            if condition.source.path in self.objectspace.valid_enums:
+                valid_enum = self.objectspace.valid_enums[condition.source.path]['values']
+                remove_param = [param_idx for param_idx, param in enumerate(condition.param) \
+                                if param.type != 'variable' and param.text not in valid_enum]
+                if not remove_param:
+                    continue
+                remove_param.sort(reverse=True)
+                for idx in remove_param:
+                    del condition.param[idx]
+                if not condition.param and condition.name.endswith('_if_not_in'):
+                    self.force_actions_to_variable(condition)
+                    remove_conditions.append(condition_idx)
+        remove_conditions.sort(reverse=True)
+        for idx in remove_conditions:
+            self.objectspace.space.constraints.condition.pop(idx)
+
+    def remove_condition_with_empty_target(self):
+        """remove condition with empty target
+        """
+        # optional target are remove, condition could be empty
+        remove_conditions = [condition_idx for condition_idx, condition in \
+                             enumerate(self.objectspace.space.constraints.condition) \
+                             if not condition.target]
+        remove_conditions.sort(reverse=True)
+        for idx in remove_conditions:
+            self.objectspace.space.constraints.condition.pop(idx)
+
+    def convert_condition(self):
+        """valid and manage <condition>
+        """
+        for condition in self.objectspace.space.constraints.condition:
+            actions = self.get_actions_from_condition(condition.name)
+            for param in condition.param:
+                for target in condition.target:
+                    leader_or_variable, variables = self._get_family_variables_from_target(target)
+                    # if option is already disable, do not apply disable_if_in
+                    # check only the first action (example of multiple actions:
+                    # 'hidden', 'frozen', 'force_default_on_freeze')
+                    main_action = actions[0]
+                    if getattr(leader_or_variable, main_action, False) is True:
+                        continue
+                    self.build_property(leader_or_variable,
+                                        param,
+                                        condition,
+                                        main_action,
+                                        )
+                    if isinstance(leader_or_variable, self.objectspace.variable) and \
+                            (leader_or_variable.auto_save or leader_or_variable.auto_freeze) and \
+                            'force_default_on_freeze' in actions:
+                        continue
+                    for action in actions[1:]:
+                        # other actions are set to the variable or children of family
+                        for variable in variables:
+                            self.build_property(variable,
+                                                param,
+                                                condition,
+                                                action,
+                                                )
+
+    def build_property(self,
+                       obj,
+                       param: 'self.objectspace.param',
+                       condition: 'self.objectspace.condition',
+                       action: str,
+                       ) -> 'self.objectspace.property_':
+        """build property_ for a condition
+        """
+        prop = self.objectspace.property_(obj.xmlfiles)
+        prop.type = 'calculation'
+        prop.inverse = condition.name.endswith('_if_not_in')
+        prop.source = condition.source
+        prop.expected = param
+        prop.name = action
+        if not hasattr(obj, 'properties'):
+            obj.properties = []
+        obj.properties.append(prop)

src/rougail/annotator/family.py

@@ -0,0 +1,300 @@
+"""Annotate family
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from ..i18n import _
+from ..error import DictConsistencyError
+from .variable import Walk
+
+
+class Mode:  # pylint: disable=R0903
+    """Class to manage mode level
+    """
+    def __init__(self,
+                 level: int,
+                 ) -> None:
+        self.level = level
+
+    def __gt__(self,
+               other: int,
+               ) -> bool:
+        return other.level < self.level
+
+
+class FamilyAnnotator(Walk):
+    """Annotate family
+    """
+    def __init__(self,
+                 objectspace,
+                 ):
+        self.objectspace = objectspace
+        if not hasattr(self.objectspace.space, 'variables'):
+            return
+        self.modes = {name: Mode(idx) for idx, name in enumerate(self.objectspace.rougailconfig['modes_level'])}
+        self.remove_empty_families()
+        self.family_names()
+        self.change_modes()
+        self.dynamic_families()
+        self.convert_help()
+
+    def _has_variable(self,
+                      family: 'self.objectspace.family',
+                      ) -> bool:
+        if hasattr(family, 'variable'):
+            for variable in family.variable.values():
+                if isinstance(variable, self.objectspace.family):
+                    if self._has_variable(variable):
+                        return True
+                else:
+                    return True
+        return False
+
+    def remove_empty_families(self) -> None:
+        """Remove all families without any variable
+        """
+        for families in self.objectspace.space.variables.values():
+            removed_families = []
+            for family_name, family in families.variable.items():
+                if isinstance(family, self.objectspace.family) and not self._has_variable(family):
+                    removed_families.append(family_name)
+            for family_name in removed_families:
+                del families.variable[family_name]
+
+    def family_names(self) -> None:
+        """Set doc, path, ... to family
+        """
+        for family in self.get_families():
+            if not hasattr(family, 'description'):
+                family.description = family.name
+            family.doc = family.description
+            del family.description
+
+    def change_modes(self):
+        """change the mode of variables
+        """
+        modes_level = self.objectspace.rougailconfig['modes_level']
+        default_variable_mode = self.objectspace.rougailconfig['default_variable_mode']
+        if default_variable_mode not in modes_level:
+            msg = _(f'default variable mode "{default_variable_mode}" is not a valid mode, '
+                    f'valid modes are {modes_level}')
+            raise DictConsistencyError(msg, 72, None)
+        default_family_mode = self.objectspace.rougailconfig['default_family_mode']
+        if default_family_mode not in modes_level:
+            msg = _(f'default family mode "{default_family_mode}" is not a valid mode, '
+                    f'valid modes are {modes_level}')
+            raise DictConsistencyError(msg, 73, None)
+        families = list(self.get_families())
+        for family in families:
+            self.valid_mode(family)
+            self._set_default_mode(family)
+        families.reverse()
+        for family in families:
+            self._change_family_mode(family)
+
+    def valid_mode(self,
+                   obj,
+                   ) -> None:
+        modes_level = self.objectspace.rougailconfig['modes_level']
+        if hasattr(obj, 'mode') and obj.mode not in modes_level:
+            msg = _(f'mode "{obj.mode}" for "{obj.name}" is not a valid mode, '
+                    f'valid modes are {modes_level}')
+            raise DictConsistencyError(msg, 71, obj.xmlfiles)
+
+    def _set_default_mode(self,
+                         family: 'self.objectspace.family',
+                         ) -> None:
+        if hasattr(family, 'mode') and 'mode' in vars(family):
+            family_mode = family.mode
+        else:
+            family_mode = None
+        if not hasattr(family, 'variable'):
+            return
+        for variable in family.variable.values():
+            self.valid_mode(variable)
+            if isinstance(variable, self.objectspace.family):
+                if family_mode and not self._has_mode(variable):
+                    self._set_auto_mode(variable, family_mode)
+                continue
+            if isinstance(variable, self.objectspace.leadership):
+                func = self._set_default_mode_leader
+            else:
+                func = self._set_default_mode_variable
+            func(variable, family_mode)
+
+    @staticmethod
+    def _has_mode(obj) -> bool:
+        return 'mode' in vars(obj) and not hasattr(obj, 'mode_auto')
+
+    def _set_default_mode_variable(self,
+                                   variable: 'self.objectspace.variable',
+                                   family_mode: str,
+                                   ) -> None:
+        # auto_save or auto_freeze variable is set to 'basic' mode
+        # if its mode is not defined by the user
+        if not self._has_mode(variable) and \
+                (variable.auto_save is True or variable.auto_freeze is True):
+            variable.mode = self.objectspace.rougailconfig['modes_level'][0]
+        # mandatory variable without value is a basic variable
+        elif not self._has_mode(variable) and \
+                variable.mandatory is True and \
+                not hasattr(variable, 'default') and \
+                not hasattr(variable, 'default_multi'):
+            variable.mode = self.objectspace.rougailconfig['modes_level'][0]
+        elif family_mode and not self._has_mode(variable):
+            self._set_auto_mode(variable, family_mode)
+
+    @staticmethod
+    def _set_auto_mode(obj, mode: str) -> None:
+        obj.mode = mode
+        obj.mode_auto = True
+
+    def _set_default_mode_leader(self,
+                                 leadership: 'self.objectspace.leadership',
+                                 family_mode: str,
+                                 ) -> None:
+        leader_mode = None
+        for follower in leadership.variable:
+            self.valid_mode(follower)
+            if follower.auto_save is True:
+                msg = _(f'leader/followers "{follower.name}" could not be auto_save')
+                raise DictConsistencyError(msg, 29, leadership.xmlfiles)
+            if follower.auto_freeze is True:
+                msg = f'leader/followers "{follower.name}" could not be auto_freeze'
+                raise DictConsistencyError(_(msg), 30, leadership.xmlfiles)
+            if leader_mode is not None:
+                if hasattr(follower, 'mode'):
+                    follower_mode = follower.mode
+                else:
+                    follower_mode = self.objectspace.rougailconfig['default_variable_mode']
+                if self.modes[leader_mode] > self.modes[follower_mode]:
+                    if self._has_mode(follower) and not self._has_mode(leadership.variable[0]):
+                        # if follower has mode but not the leader
+                        self._set_auto_mode(leadership.variable[0], follower_mode)
+                    else:
+                        # leader's mode is minimum level
+                        if self._has_mode(follower):
+                            msg = _(f'the follower "{follower.name}" is in "{follower_mode}" mode '
+                                    f'but leader have the higher mode "{leader_mode}"')
+                            raise DictConsistencyError(msg, 63, follower.xmlfiles)
+                        self._set_auto_mode(follower, leader_mode)
+            self._set_default_mode_variable(follower,
+                                            family_mode,
+                                            )
+            if leader_mode is None:
+                if hasattr(leadership.variable[0], 'mode'):
+                    leader_mode = leadership.variable[0].mode
+                else:
+                    leader_mode = self.objectspace.rougailconfig['default_variable_mode']
+        if hasattr(leadership.variable[0], 'mode'):
+            leader_mode = leadership.variable[0].mode
+        self._set_auto_mode(leadership, leader_mode)
+
+    def _change_family_mode(self,
+                           family: 'self.objectspace.family',
+                           ) -> None:
+        if hasattr(family, 'mode'):
+            family_mode = family.mode
+        else:
+            family_mode = self.objectspace.rougailconfig['default_family_mode']
+        min_variable_mode = self.objectspace.rougailconfig['modes_level'][-1]
+        # change variable mode, but not if variables are not in a family
+        if hasattr(family, 'variable'):
+            for variable in family.variable.values():
+                if not isinstance(variable, self.objectspace.family):
+                    if isinstance(variable, self.objectspace.leadership):
+                        func = self._change_variable_mode_leader
+                    else:
+                        func = self._change_variable_mode
+                    func(variable,
+                         family_mode,
+                         )
+                elif not hasattr(variable, 'mode'):
+                    variable.mode = self.objectspace.rougailconfig['default_family_mode']
+                if self.modes[min_variable_mode] > self.modes[variable.mode]:
+                    min_variable_mode = variable.mode
+        if isinstance(family, self.objectspace.family) and \
+                (not hasattr(family, 'mode') or family.mode != min_variable_mode):
+            # set the lower variable mode to family
+            if self._has_mode(family):
+                msg = _(f'the family "{family.name}" is in "{family.mode}" mode but variables and '
+                        f'families inside have the higher modes "{min_variable_mode}"')
+                raise DictConsistencyError(msg, 62, family.xmlfiles)
+            self._set_auto_mode(family, min_variable_mode)
+
+    def _change_variable_mode(self,
+                             variable,
+                             family_mode: str,
+                             ) -> None:
+        if hasattr(variable, 'mode'):
+            variable_mode = variable.mode
+        else:
+            variable_mode = self.objectspace.rougailconfig['default_variable_mode']
+        # none basic variable in high level family has to be in high level
+        if self.modes[variable_mode] < self.modes[family_mode]:
+            if self._has_mode(variable):
+                msg = _(f'the variable "{variable.name}" is in "{variable_mode}" mode '
+                        f'but family has the higher family mode "{family_mode}"')
+                raise DictConsistencyError(msg, 61, variable.xmlfiles)
+            self._set_auto_mode(variable, family_mode)
+        if not hasattr(variable, 'mode'):
+            variable.mode = variable_mode
+
+    def _change_variable_mode_leader(self,
+                                     leadership,
+                                     family_mode: str,
+                                     ) -> None:
+        for follower in leadership.variable:
+            self._change_variable_mode(follower,
+                                      family_mode,
+                                      )
+        leadership.variable[0].mode = None
+
+    def dynamic_families(self):
+        """link dynamic families to object
+        """
+        for family in self.get_families():
+            if 'dynamic' not in vars(family):
+                continue
+            family.suffixes = self.objectspace.paths.get_variable(family.dynamic)
+            del family.dynamic
+            if not family.suffixes.multi:
+                msg = _(f'dynamic family "{family.name}" must be linked '
+                        f'to multi variable')
+                raise DictConsistencyError(msg, 16, family.xmlfiles)
+            for variable in family.variable.values():
+                if isinstance(variable, self.objectspace.family):
+                    msg = _(f'dynamic family "{family.name}" cannot contains another family')
+                    raise DictConsistencyError(msg, 22, family.xmlfiles)
+
+    def convert_help(self):
+        """Convert variable help
+        """
+        for family in self.get_families():
+            if not hasattr(family, 'help'):
+                continue
+            if not hasattr(family, 'information'):
+                family.information = self.objectspace.information(family.xmlfiles)
+            family.information.help = family.help
+            del family.help

src/rougail/annotator/fill.py

@@ -0,0 +1,93 @@
+"""Fill annotator
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from ..utils import load_modules
+from ..i18n import _
+from ..error import DictConsistencyError
+
+from .target import TargetAnnotator
+from .param import ParamAnnotator
+
+
+CALC_MULTI = ('calc_value', 'calc_list', 'get_range', 'calc_val_first_value', 'unbound_filename')
+
+
+class FillAnnotator(TargetAnnotator, ParamAnnotator):
+    """Fill annotator
+    """
+    def __init__(self,
+                 objectspace,
+                 eosfunc_file,
+                 ):
+        self.objectspace = objectspace
+        if not hasattr(objectspace.space, 'constraints') or \
+                not hasattr(self.objectspace.space.constraints, 'fill'):
+            return
+        self.functions = dir(load_modules(eosfunc_file))
+        self.functions.extend(self.objectspace.rougailconfig['internal_functions'])
+        self.target_is_uniq = True
+        self.only_variable = True
+        self.allow_function = False
+        self.convert_target(self.objectspace.space.constraints.fill)
+        self.convert_param(self.objectspace.space.constraints.fill)
+        self.fill_to_value()
+        del self.objectspace.space.constraints.fill
+
+    def calc_is_multi(self, variable: 'self.objectspace.variable') -> bool:
+        multi = variable.multi
+        if multi is False:
+            return multi
+        if multi == 'submulti':
+            return True
+        return not self.objectspace.paths.is_follower(variable.path)
+
+    def fill_to_value(self) -> None:
+        """valid and manage <fill>
+        """
+        for fill in self.objectspace.space.constraints.fill:
+            for target in fill.target:
+                # test if the function exists
+                if fill.name not in self.functions:
+                    msg = _(f'cannot find fill function "{fill.name}"')
+                    raise DictConsistencyError(msg, 25, fill.xmlfiles)
+
+                # create an object value
+                value = self.objectspace.value(fill.xmlfiles)
+                value.type = 'calculation'
+                value.name = fill.name
+                if fill.name not in CALC_MULTI:
+                    is_calc_multi = self.calc_is_multi(target.name)
+                else:
+                    is_calc_multi = False
+                value.calc_multi = is_calc_multi
+                if target.name.namespace == 'services':
+                    target.name.default = value
+                else:
+                    target.name.value = [value]
+
+                # manage params
+                if hasattr(fill, 'param') and fill.param:
+                    value.param = fill.param

src/rougail/annotator/group.py

@@ -0,0 +1,160 @@
+"""Annotate group
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from ..i18n import _
+from ..error import DictConsistencyError
+from ..utils import normalize_family
+
+
+class GroupAnnotator:
+    """Annotate group
+    """
+    def __init__(self,
+                 objectspace,
+                 ):
+        self.objectspace = objectspace
+        if not hasattr(self.objectspace.space, 'constraints') or \
+                not hasattr(self.objectspace.space.constraints, 'group'):
+            return
+        self.convert_groups()
+
+    def convert_groups(self):  # pylint: disable=C0111
+        """convert groups
+        """
+        # store old leaders family name
+        cache_paths = {}
+        for group in self.objectspace.space.constraints.group:
+            if group.leader in cache_paths:
+                leader_fam_path = cache_paths[group.leader]
+            else:
+                leader_fam_path = self.objectspace.paths.get_variable_family_path(group.leader, group.xmlfiles)
+                cache_paths[group.leader] = leader_fam_path
+            follower_names = list(group.follower.keys())
+            leader = self.objectspace.paths.get_variable(group.leader)
+            if '.' not in leader_fam_path:
+                # it's a namespace
+                ori_leader_family = self.objectspace.space.variables[leader_fam_path]
+            else:
+                # it's a sub family
+                ori_leader_family = self.objectspace.paths.get_family(leader_fam_path,
+                                                                      leader.namespace,
+                                                                      )
+            has_a_leader = False
+            for variable in list(ori_leader_family.variable.values()):
+                if isinstance(variable, self.objectspace.leadership) and \
+                        variable.variable[0].name == leader.name:
+                    # append follower to an existed leadership
+                    leader_space = variable
+                    has_a_leader = True
+                elif variable.name == leader.name:
+                    # it's a leader
+                    leader_space = self.manage_leader(variable,
+                                                      group,
+                                                      ori_leader_family,
+                                                      )
+                    has_a_leader = True
+                elif has_a_leader:
+                    # it's should be a follower
+                    self.manage_follower(follower_names.pop(0),
+                                         leader_fam_path,
+                                         variable,
+                                         leader_space,
+                                         )
+                    # this variable is not more in ori_leader_family
+                    ori_leader_family.variable.pop(normalize_family(variable.name))
+                    if follower_names == []:
+                        # no more follower
+                        break
+            else:
+                joined = '", "'.join(follower_names)
+                msg = _(f'when parsing leadership, we espect to find those followers "{joined}"')
+                raise DictConsistencyError(msg, 31, variable.xmlfiles)
+        del self.objectspace.space.constraints.group
+
+    def manage_leader(self,
+                      variable: 'Variable',
+                      group: 'Group',
+                      ori_leader_family,
+                      ) -> 'Leadership':
+        """manage leader's variable
+        """
+        if variable.multi is not True:
+            msg = _(f'the variable "{variable.name}" in a group must be multi')
+            raise DictConsistencyError(msg, 32, variable.xmlfiles)
+        if hasattr(group, 'name'):
+            ori_leadership_name = group.name
+        else:
+            ori_leadership_name = variable.name
+        leadership_name = normalize_family(ori_leadership_name)
+        leader_space = self.objectspace.leadership(variable.xmlfiles)
+        leader_space.variable = []
+        leader_space.name = leadership_name
+        leader_space.hidden = variable.hidden
+        if variable.hidden:
+            variable.frozen = True
+            variable.force_default_on_freeze = True
+        variable.hidden = None
+        if hasattr(group, 'description'):
+            leader_space.doc = group.description
+        elif variable.name == leadership_name and hasattr(variable, 'description'):
+            leader_space.doc = variable.description
+        else:
+            leader_space.doc = ori_leadership_name
+        leadership_path = ori_leader_family.path + '.' + leadership_name
+        self.objectspace.paths.add_leadership(variable.namespace,
+                                              leadership_path,
+                                              leader_space,
+                                              )
+        ori_leader_family.variable[normalize_family(variable.name)] = leader_space
+        leader_space.variable.append(variable)
+        self.objectspace.paths.set_leader(variable.namespace,
+                                          ori_leader_family.path,
+                                          leadership_name,
+                                          normalize_family(variable.name),
+                                          )
+        return leader_space
+
+    def manage_follower(self,
+                        follower_name: str,
+                        leader_family_name: str,
+                        variable: 'Variable',
+                        leader_space: 'Leadership',
+                        ) -> None:
+        """manage follower
+        """
+        if variable.name != follower_name:
+            msg = _('when parsing leadership, we expect to find the follower '
+                    f'"{follower_name}" but we found "{variable.name}"')
+            raise DictConsistencyError(msg, 33, variable.xmlfiles)
+        self.objectspace.paths.set_leader(variable.namespace,
+                                          leader_family_name,
+                                          leader_space.name,
+                                          normalize_family(variable.name),
+                                          )
+        if leader_space.hidden:
+            variable.frozen = True
+            variable.force_default_on_freeze = True
+        leader_space.variable.append(variable)

src/rougail/annotator/param.py

@@ -0,0 +1,147 @@
+"""Param annotator
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+try:
+    import tiramisu3 as tiramisu
+except ModuleNotFoundError:
+    import tiramisu
+
+from .variable import CONVERT_OPTION
+
+from ..i18n import _
+from ..error import DictConsistencyError
+
+
+class ParamAnnotator:
+    """Param annotator
+    """
+    objectspace = None
+
+    @staticmethod
+    def valid_type_validation(obj) -> None:
+        """Function to valid type (redefine in fill/condition/check classes)
+        """
+        return None
+
+    def convert_param(self, objects) -> None:
+        """ valid and convert param
+        """
+        for obj in objects:
+            if not hasattr(obj, 'param'):
+                continue
+            param_to_delete = []
+            variable_type = self.valid_type_validation(obj)
+            for param_idx, param in enumerate(obj.param):
+                if hasattr(param, 'text'):
+                    if param.type in ['suffix', 'index']:
+                        msg = _(f'"{param.type}" parameter must not have a value')
+                        raise DictConsistencyError(msg, 28, obj.xmlfiles)
+                    if param.type == 'nil':
+                        if param.text is not None:
+                            msg = _(f'"{param.type}" parameter must not have a value')
+                            raise DictConsistencyError(msg, 40, obj.xmlfiles)
+                    elif param.type == 'variable':
+                        try:
+                            path, suffix = self.objectspace.paths.get_variable_path(param.text,
+                                                                                    obj.namespace,
+                                                                                    param.xmlfiles,
+                                                                                    )
+                            param.text = self.objectspace.paths.get_variable(path)
+                            if variable_type and param.text.type != variable_type:
+                                msg = _(f'"{obj.name}" has type "{variable_type}" but param '
+                                        f'has type "{param.text.type}"')
+                                raise DictConsistencyError(msg, 26, param.xmlfiles)
+                            if suffix:
+                                param.suffix = suffix
+                                family_path = self.objectspace.paths.get_variable_family_path(path)
+                                namespace = param.text.namespace
+                                param.family = self.objectspace.paths.get_family(family_path,
+                                                                                 namespace,
+                                                                                 )
+                        except DictConsistencyError as err:
+                            if err.errno != 42 or not param.optional:
+                                raise err
+                            param_to_delete.append(param_idx)
+                    elif param.type == 'function':
+                        if not self.allow_function:
+                            msg = _(f'cannot use "function" type')
+                            raise DictConsistencyError(msg, 74, param.xmlfiles)
+                        if not param.text in self.functions:
+                            msg = _(f'cannot find function "{param.text}"')
+                            raise DictConsistencyError(msg, 67, param.xmlfiles)
+                        if param_idx != 0:
+                            msg = _(f'function "{param.text}" must only set has first parameter')
+                            raise DictConsistencyError(msg, 75, param.xmlfiles)
+                    elif variable_type:
+                        self._convert_with_variable_type(variable_type, param)
+                    continue
+                # no param.text
+                if param.type == 'suffix':
+                    for target in obj.target:
+                        if not self.objectspace.paths.variable_is_dynamic(target.name.path):
+                            msg = _(f'"{param.type}" parameter cannot be set with target '
+                                    f'"{target.name}" which is not a dynamic variable')
+                            raise DictConsistencyError(msg, 53, obj.xmlfiles)
+                elif param.type == 'index':
+                    for target in obj.target:
+                        if not self.objectspace.paths.is_follower(target.name.path):
+                            msg = _(f'"{param.type}" parameter cannot be set with target '
+                                    f'"{target.name.name}" which is not a follower variable')
+                            raise DictConsistencyError(msg, 60, obj.xmlfiles)
+                elif param.type == 'nil':
+                    param.text = None
+                elif param.type == 'string':
+                    param.text = ''
+                    if variable_type:
+                        self._convert_with_variable_type(variable_type, param)
+                else:
+                    msg = _(f'"{param.type}" parameter must have a value')
+                    raise DictConsistencyError(msg, 27, obj.xmlfiles)
+            param_to_delete.sort(reverse=True)
+            for param_idx in param_to_delete:
+                obj.param.pop(param_idx)
+
+    @staticmethod
+    def _convert_with_variable_type(variable_type: str,
+                                    param: 'self.objectspace.param',
+                                    ) -> None:
+        if 'type' in vars(param) and variable_type != param.type:
+            msg = _(f'parameter has incompatible type "{param.type}" '
+                    f'with type "{variable_type}"')
+            raise DictConsistencyError(msg, 7, param.xmlfiles)
+        try:
+            option = CONVERT_OPTION[variable_type]
+            param.text = option.get('func', str)(param.text)
+            getattr(tiramisu, option['opttype'])('test',
+                                                 'Object to valid value',
+                                                 param.text,
+                                                 **option.get('initkwargs', {}),
+                                                 )
+        except ValueError as err:
+            msg = _(f'unable to change type of value "{param.text}" '
+                    f'is not a valid "{variable_type}"')
+            raise DictConsistencyError(msg, 13, param.xmlfiles) from err
+        param.type = variable_type

src/rougail/annotator/property.py

@@ -0,0 +1,108 @@
+"""Annotate properties
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from ..i18n import _
+from ..error import DictConsistencyError
+from .variable import Walk
+
+
+PROPERTIES = ('hidden', 'frozen', 'force_default_on_freeze',
+              'force_store_value', 'disabled', 'mandatory')
+
+
+class PropertyAnnotator(Walk):
+    """Annotate properties
+    """
+    def __init__(self, objectspace):
+        self.objectspace = objectspace
+        if hasattr(self.objectspace.space, 'services'):
+            self.convert_services()
+        if hasattr(self.objectspace.space, 'variables'):
+            self.convert_family()
+            self.convert_variable()
+
+    def convert_property(self,
+                         variable,
+                         ) -> None:
+        """convert properties
+        """
+        # hidden variable is also frozen
+        if isinstance(variable, self.objectspace.variable) and variable.hidden is True:
+            if not variable.auto_freeze:
+                variable.frozen = True
+            if not variable.auto_save and \
+                    not variable.auto_freeze and \
+                    'force_default_on_freeze' not in vars(variable):
+                variable.force_default_on_freeze = True
+        if not hasattr(variable, 'properties'):
+            variable.properties = []
+        for prop in PROPERTIES:
+            if hasattr(variable, prop):
+                if getattr(variable, prop) is True:
+#                    for subprop in CONVERT_PROPERTIES.get(prop, [prop]):
+                    variable.properties.append(prop)
+                setattr(variable, prop, None)
+        if hasattr(variable, 'mode') and variable.mode:
+            variable.properties.append(variable.mode)
+            variable.mode = None
+        if 'force_store_value' in variable.properties and \
+                'force_default_on_freeze' in variable.properties:  # pragma: no cover
+            # should not appened
+            msg = _('cannot have auto_freeze or auto_save with the hidden '
+                    f'variable "{variable.name}"')
+            raise DictConsistencyError(msg, 50, variable.xmlfiles)
+        if not variable.properties:
+            del variable.properties
+
+    def convert_services(self) -> None:
+        """convert services
+        """
+        self.convert_property(self.objectspace.space.services)
+        for services in self.objectspace.space.services.service.values():
+            self.convert_property(services)
+            for service in vars(services).values():
+                if not isinstance(service, self.objectspace.family):
+                    continue
+                self.convert_property(service)
+                for family in service.family:
+                    self.convert_property(family)
+                    for variable in family.variable:
+                        self.convert_property(variable)
+
+    def convert_family(self) -> None:
+        """convert families
+        """
+        for family in self.get_families():
+            self.convert_property(family)
+
+    def convert_variable(self) -> None:
+        """convert variables
+        """
+        for variable in self.get_variables(with_leadership=True):
+            if isinstance(variable, self.objectspace.leadership):
+                for follower in variable.variable:
+                    self.convert_property(follower)
+            self.convert_property(variable)

src/rougail/annotator/service.py

@@ -0,0 +1,303 @@
+"""Annotate services
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from os.path import basename
+from typing import Tuple
+
+from ..i18n import _
+from ..utils import normalize_family
+from ..error import DictConsistencyError
+from ..config import RougailConfig
+# a object's attribute has some annotations
+# that shall not be present in the exported (flatened) XML
+ERASED_ATTRIBUTES = ('redefine', 'exists', 'optional', 'remove_check', 'namespace',
+                     'remove_condition', 'path', 'instance_mode', 'index',
+                     'level', 'remove_fill', 'xmlfiles', 'type', 'reflector_name',
+                     'reflector_object',)
+ALLOW_ATTRIBUT_NOT_MANAGE = ['file']
+
+
+class ServiceAnnotator:
+    """Manage service's object
+    for example::
+        <services>
+            <service name="test">
+                <service_access service='ntp'>
+                </service_access>
+            </service>
+        </services>
+    """
+    def __init__(self, objectspace):
+        self.objectspace = objectspace
+        self.uniq_overrides = []
+        if 'network_type' not in self.objectspace.types:
+            self.objectspace.types['network_type'] = self.objectspace.types['ip_type']
+        if hasattr(self.objectspace.space, 'services'):
+            if not hasattr(self.objectspace.space.services, 'service'):
+                del self.objectspace.space.services
+            else:
+                self.convert_services()
+
+    def convert_services(self):
+        """convert services to variables
+        """
+        self.objectspace.space.services.hidden = True
+        self.objectspace.space.services.name = 'services'
+        self.objectspace.space.services.doc = 'services'
+        self.objectspace.space.services.path = 'services'
+        for service_name, service in self.objectspace.space.services.service.items():
+            activate_obj = self._generate_element('boolean',
+                                                  None,
+                                                  None,
+                                                  'activate',
+                                                  True,
+                                                  service,
+                                                  '.'.join(['services', normalize_family(service_name), 'activate']),
+                                                  )
+            for elttype, values in dict(vars(service)).items():
+                if elttype == 'servicelist':
+                    self.objectspace.list_conditions.setdefault('servicelist',
+                                                                {}).setdefault(
+                                                                    values,
+                                                                    []).append(activate_obj)
+                    continue
+                if not isinstance(values, (dict, list)) or elttype in ERASED_ATTRIBUTES:
+                    continue
+                if not service.manage and elttype not in ALLOW_ATTRIBUT_NOT_MANAGE:
+                    msg = _(f'unmanage service cannot have "{elttype}"')
+                    raise DictConsistencyError(msg, 66, service.xmlfiles)
+                if elttype != 'ip':
+                    eltname = elttype + 's'
+                else:
+                    eltname = elttype
+                path = '.'.join(['services', normalize_family(service_name), eltname])
+                family = self._gen_family(eltname,
+                                          path,
+                                          service.xmlfiles,
+                                          )
+                if isinstance(values, dict):
+                    values = list(values.values())
+                family.family = self.make_group_from_elts(service_name,
+                                                          elttype,
+                                                          values,
+                                                          path,
+                                                          )
+                setattr(service, elttype, family)
+            manage = self._generate_element('boolean',
+                                            None,
+                                            None,
+                                            'manage',
+                                            service.manage,
+                                            service,
+                                            '.'.join(['services', normalize_family(service_name), 'manage']),
+                                                  )
+            service.variable = [activate_obj, manage]
+            service.doc = service.name
+
+    def make_group_from_elts(self,
+                             service_name,
+                             elttype,
+                             elts,
+                             path,
+                             ):
+        """Splits each objects into a group (and `OptionDescription`, in tiramisu terms)
+        and build elements and its attributes (the `Options` in tiramisu terms)
+        """
+        families = []
+        listname = '{}list'.format(elttype)
+        for elt in elts:
+            # try to launch _update_xxxx() function
+            update_elt = '_update_' + elttype
+            if hasattr(self, update_elt):
+                getattr(self, update_elt)(elt,
+                                          service_name,
+                                          )
+            c_name, subpath = self._get_name_path(elt,
+                                                  path,
+                                                  )
+            family = self._gen_family(c_name,
+                                      subpath,
+                                      elt.xmlfiles,
+                                      )
+            family.variable = []
+            activate_obj = self._generate_element('boolean',
+                                                  None,
+                                                  None,
+                                                  'activate',
+                                                  True,
+                                                  elt,
+                                                  '.'.join([subpath, 'activate']),
+                                                  )
+            for key in dir(elt):
+                if key.startswith('_') or key.endswith('_type') or key in ERASED_ATTRIBUTES:
+                    continue
+                value = getattr(elt, key)
+                if key == listname:
+                    self.objectspace.list_conditions.setdefault(listname,
+                                                                {}).setdefault(
+                                                                    value,
+                                                                    []).append(activate_obj)
+                    continue
+                if key == 'name':
+                    dtd_key_type = elttype + '_type'
+                else:
+                    dtd_key_type = key + '_type'
+                elt_type = getattr(elt, dtd_key_type, 'string')
+                if elt_type == 'variable':
+                    elt_type = 'symlink'
+                family.variable.append(self._generate_element(elt_type,
+                                                              dtd_key_type,
+                                                              elttype,
+                                                              key,
+                                                              value,
+                                                              elt,
+                                                              f'{subpath}.{key}'
+                                                              ))
+            family.variable.append(activate_obj)
+            families.append(family)
+        return families
+
+    def _get_name_path(self,
+                       elt,
+                       path: str,
+                       ) -> Tuple[str, str]:
+        # create element name, if already exists, add _xx to be uniq
+        if hasattr(elt, 'source') and elt.source:
+            name = elt.source
+        else:
+            name = elt.name
+        idx = 0
+        while True:
+            c_name = name
+            if idx:
+                c_name += f'_{idx}'
+            subpath = '{}.{}'.format(path, normalize_family(c_name))
+            try:
+                self.objectspace.paths.get_family(subpath, 'services')
+            except DictConsistencyError as err:
+                if err.errno == 42:
+                    return c_name, subpath
+            idx += 1
+
+    def _gen_family(self,
+                    name,
+                    path,
+                    xmlfiles
+                    ):
+        family = self.objectspace.family(xmlfiles)
+        family.name = normalize_family(name)
+        family.doc = name
+        family.mode = None
+        self.objectspace.paths.add_family('services',
+                                          path,
+                                          family,
+                                          None,
+                                          )
+        return family
+
+    def _generate_element(self,
+                          type_,
+                          dtd_key_type,
+                          elttype,
+                          key,
+                          value,
+                          elt,
+                          path,
+                          ):  # pylint: disable=R0913
+        variable = self.objectspace.variable(elt.xmlfiles)
+        variable.name = normalize_family(key)
+        variable.mode = None
+        variable.type = type_
+        if type_ == 'symlink':
+            variable.opt = self.objectspace.paths.get_variable(value)
+            variable.multi = None
+            needed_type = self.objectspace.types[dtd_key_type]
+            if needed_type not in ('variable', variable.opt.type):
+                msg = _(f'"{value}" in "{elttype}" must be a variable with type '
+                        f'"{needed_type}" not "{variable.opt.type}"')
+                raise DictConsistencyError(msg, 58, elt.xmlfiles)
+
+        else:
+            variable.doc = key
+            variable.default = value
+            variable.namespace = 'services'
+        self.objectspace.paths.add_variable('services',
+                                            path,
+                                            'service',
+                                            False,
+                                            variable,
+                                            )
+        return variable
+
+    def _update_override(self,
+                         override,
+                         service_name,
+                         ):
+
+        if service_name in self.uniq_overrides:
+            msg = _('only one override is allowed by service, '
+                    'please use a variable multiple if you want have more than one IP')
+            raise DictConsistencyError(msg, 69, override.xmlfiles)
+        self.uniq_overrides.append(service_name)
+        override.name = service_name
+        if not hasattr(override, 'engine'):
+            override.engine = RougailConfig['default_engine']
+        if not hasattr(override, 'source'):
+            override.source = f'{service_name}.service'
+
+    @staticmethod
+    def _update_file(file_,
+                     service_name,
+                     ):
+        if not hasattr(file_, 'file_type') or file_.file_type == "filename":
+            if not hasattr(file_, 'source'):
+                file_.source = basename(file_.name)
+        elif not hasattr(file_, 'source'):
+            msg = _(f'attribute "source" is mandatory for the file "{file_.name}" '
+                    f'"({service_name})"')
+            raise DictConsistencyError(msg, 34, file_.xmlfiles)
+        if not hasattr(file_, 'engine'):
+            file_.engine = RougailConfig['default_engine']
+
+    def _update_ip(self,
+                   ip,
+                   service_name,
+                   ) -> None:
+        variable = self.objectspace.paths.get_variable(ip.name, ip.xmlfiles)
+        if variable.type not in ['ip', 'network', 'network_cidr']:
+            msg = _(f'ip cannot be linked to "{variable.type}" variable "{ip.name}"')
+            raise DictConsistencyError(msg, 70, ip.xmlfiles)
+        if variable.type in ['ip', 'network_cidr'] and hasattr(ip, 'netmask'):
+            msg = _(f'ip with ip_type "{variable.type}" must not have netmask')
+            raise DictConsistencyError(msg, 59, ip.xmlfiles)
+        if variable.type == 'network' and not hasattr(ip, 'netmask'):
+            msg = _(f'ip with ip_type "{variable.type}" must have netmask')
+            raise DictConsistencyError(msg, 64, ip.xmlfiles)
+        if hasattr(ip, 'netmask'):
+            netmask = self.objectspace.paths.get_variable(ip.netmask, ip.xmlfiles)
+            if netmask.type != 'netmask':
+                msg = _(f'netmask in ip must have type "netmask", not "{netmask.type}"')
+                raise DictConsistencyError(msg, 65, ip.xmlfiles)

src/rougail/annotator/target.py

@@ -0,0 +1,91 @@
+"""Target annotator
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from ..i18n import _
+from ..error import DictConsistencyError
+
+
+class TargetAnnotator:
+    """Target annotator
+    """
+    def convert_target(self,
+                       objects,
+                       ) -> None:
+        """ valid and convert param
+        """
+        targets = []
+        for obj in objects:
+            if not hasattr(obj, 'target'):
+                msg = _('target is mandatory')
+                raise DictConsistencyError(msg, 9, obj.xmlfiles)
+            remove_targets = []
+            for index, target in enumerate(obj.target):
+                # test if it's redefined calculation
+                if self.target_is_uniq and target.name in targets:
+                    msg = _(f'A fill already exists for the target of "{target.name}" created')
+                    raise DictConsistencyError(msg, 24, obj.xmlfiles)
+                targets.append(target.name)
+
+                # let's replace the target by the path
+                try:
+                    if target.type == 'variable':
+                        path, suffix = self.objectspace.paths.get_variable_path(target.name,
+                                                                                obj.namespace,
+                                                                                )
+                        target.name = self.objectspace.paths.get_variable(path)
+                        if suffix:
+                            msg = _(f'target to {target.name.path} with suffix is not allowed')
+                            raise DictConsistencyError(msg, 35, obj.xmlfiles)
+                    elif self.only_variable:
+                        msg = _(f'target to "{target.name}" with param type "{target.type}" '
+                                f'is not allowed')
+                        raise DictConsistencyError(msg, 8, obj.xmlfiles)
+                    if target.type == 'family':
+                        if obj.name not in ['disabled_if_in',
+                                            'disabled_if_not_in',
+                                            'hidden_if_in',
+                                            'hidden_if_not_in',
+                                            ]:
+                            msg = _(f'target "{target.type}" not allow with "{obj.name}"')
+                            raise DictConsistencyError(msg, 51, target.xmlfiles)
+                        target.name = self.objectspace.paths.get_family(target.name,
+                                                                        obj.namespace,
+                                                                        )
+                    elif target.type.endswith('list') and \
+                            obj.name not in ['disabled_if_in', 'disabled_if_not_in']:
+                        msg = _(f'target "{target.type}" not allow with "{obj.name}"')
+                        raise DictConsistencyError(msg, 10, target.xmlfiles)
+                except DictConsistencyError as err:
+                    if err.errno != 42:
+                        raise err
+                    # for optional variable
+                    if not target.optional:
+                        msg = f'cannot found target "{target.type}" "{target.name}"'
+                        raise DictConsistencyError(_(msg), 12, target.xmlfiles) from err
+                    remove_targets.append(index)
+            remove_targets.sort(reverse=True)
+            for index in remove_targets:
+                obj.target.pop(index)

src/rougail/annotator/value.py

@@ -0,0 +1,98 @@
+"""Annotate value
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from .variable import Walk
+
+from ..i18n import _
+from ..error import DictConsistencyError
+
+class ValueAnnotator(Walk):  # pylint: disable=R0903
+    """Annotate value
+    """
+    def __init__(self,
+                 objectspace,
+                 ) -> None:
+        if not hasattr(objectspace.space, 'variables'):
+            return
+        self.objectspace = objectspace
+        self.convert_value()
+
+    def convert_value(self) -> None:
+        """convert value
+        """
+        for variable in self.get_variables(with_leadership=True):
+            if isinstance(variable, self.objectspace.leadership):
+                variable_type = 'leader'
+                for follower in variable.variable:
+                    self._convert_value(follower,
+                                        variable_type,
+                                        )
+                    variable_type = 'follower'
+            else:
+                self._convert_value(variable)
+
+    def _convert_value(self,
+                       variable,
+                       variable_type: str=None,
+                       ) -> None:
+        # a boolean must have value, the default value is "True"
+        if not hasattr(variable, 'value') and variable.type == 'boolean':
+            new_value = self.objectspace.value(variable.xmlfiles)
+            new_value.name = True
+            new_value.type = 'boolean'
+            variable.value = [new_value]
+        # if the variable is mandatory and doesn't have any value
+        # then the variable's mode is set to 'basic'
+        # variable with default value is mandatory
+        if hasattr(variable, 'value') and variable.value:
+            has_value = True
+            for value in variable.value:
+                if value.type == 'calculation':
+                    has_value = False
+                    break
+            if has_value and 'mandatory' not in vars(variable):
+                # if has value without any calculation
+                variable.mandatory = True
+        if not hasattr(variable, 'value'):
+            return
+        if variable.value[0].type == 'calculation':
+            variable.default = variable.value[0]
+        else:
+            if variable.multi:
+                if variable_type != 'follower':
+                    variable.default = [value.name for value in variable.value]
+                if variable_type != 'leader':
+                    if variable.multi == 'submulti':
+                        variable.default_multi = [value.name for value in variable.value]
+                    else:
+                        variable.default_multi = variable.value[0].name
+            else:
+                if len(variable.value) > 1:
+                    msg = _(f'the non multi variable "{variable.name}" cannot have '
+                            'more than one value')
+                    raise DictConsistencyError(msg, 68, variable.xmlfiles)
+                variable.default = variable.value[0].name
+        del variable.value

src/rougail/annotator/variable.py

@@ -0,0 +1,240 @@
+"""Annotate variable
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+
+from ..i18n import _
+from ..error import DictConsistencyError
+from ..objspace import convert_boolean
+
+
+CONVERT_OPTION = {'number': dict(opttype="IntOption", func=int),
+                  'float': dict(opttype="FloatOption", func=float),
+                  'choice': dict(opttype="ChoiceOption"),
+                  'string': dict(opttype="StrOption"),
+                  'password': dict(opttype="PasswordOption"),
+                  'mail': dict(opttype="EmailOption"),
+                  'boolean': dict(opttype="BoolOption", func=convert_boolean),
+                  'symlink': dict(opttype="SymLinkOption"),
+                  'filename': dict(opttype="FilenameOption"),
+                  'date': dict(opttype="DateOption"),
+                  'unix_user': dict(opttype="UsernameOption"),
+                  'ip': dict(opttype="IPOption", initkwargs={'allow_reserved': True}),
+                  'local_ip': dict(opttype="IPOption", initkwargs={'private_only': True,
+                                   'warnings_only': True}),
+                  'netmask': dict(opttype="NetmaskOption"),
+                  'network': dict(opttype="NetworkOption"),
+                  'broadcast': dict(opttype="BroadcastOption"),
+                  'netbios': dict(opttype="DomainnameOption", initkwargs={'type': 'netbios',
+                                  'warnings_only': True}),
+                  'domainname': dict(opttype="DomainnameOption", initkwargs={'type': 'domainname',
+                                 'allow_ip': False}),
+                  'hostname': dict(opttype="DomainnameOption", initkwargs={'type': 'hostname',
+                                   'allow_ip': False}),
+                  'web_address': dict(opttype="URLOption", initkwargs={'allow_ip': True,
+                                      'allow_without_dot': True}),
+                  'port': dict(opttype="PortOption", initkwargs={'allow_private': True}),
+                  'mac': dict(opttype="MACOption"),
+                  'cidr': dict(opttype="IPOption", initkwargs={'cidr': True}),
+                  'network_cidr': dict(opttype="NetworkOption", initkwargs={'cidr': True}),
+                  }
+
+
+FORCE_CHOICE = {'schedule': ['none', 'daily', 'weekly', 'monthly'],
+                'schedulemod': ['pre', 'post'],
+                }
+
+
+class Walk:
+    """Walk to objectspace to find variable or family
+    """
+    objectspace = None
+
+    def get_variables(self,
+                      with_leadership: bool=False,
+                      ):
+        """Iter all variables from the objectspace
+        """
+        for family in self.objectspace.space.variables.values():
+            yield from self._get_variables(family, with_leadership)
+
+    def _get_variables(self,
+                       family: 'self.objectspace.family',
+                       with_leadership: bool
+                       ):
+        if hasattr(family, 'variable'):
+            for variable in family.variable.values():
+                if isinstance(variable, self.objectspace.family):
+                    yield from self._get_variables(variable, with_leadership)
+                    continue
+                if not with_leadership and isinstance(variable, self.objectspace.leadership):
+                    for follower in variable.variable:
+                        yield follower
+                    continue
+                yield variable
+
+    def get_families(self):
+        """Iter all families from the objectspace
+        """
+        for family in self.objectspace.space.variables.values():
+            yield from self._get_families(family)
+
+    def _get_families(self,
+                      family: 'self.objectspace.family',
+                      ):
+        yield family
+        if hasattr(family, 'variable'):
+            for fam in family.variable.values():
+                if isinstance(fam, self.objectspace.family):
+                    yield from self._get_families(fam)
+
+
+class VariableAnnotator(Walk):  # pylint: disable=R0903
+    """Annotate variable
+    """
+    def __init__(self,
+                 objectspace,
+                 ):
+        if not hasattr(objectspace.space, 'variables'):
+            return
+        self.objectspace = objectspace
+        self.forbidden_name = ['services', self.objectspace.rougailconfig['variable_namespace']]
+        for extra in self.objectspace.rougailconfig['extra_dictionaries']:
+            self.forbidden_name.append(extra)
+        self.convert_variable()
+        self.convert_test()
+        self.convert_help()
+
+    def convert_variable(self):
+        """convert variable
+        """
+        for variable in self.get_variables(with_leadership=True):
+            if isinstance(variable, self.objectspace.leadership):
+                # first variable is a leader, others are follower
+                variable_type = 'leader'
+                for follower in variable.variable:
+                    self._convert_variable(follower,
+                                           variable_type,
+                                           )
+                    variable_type = 'follower'
+            else:
+                self._convert_variable(variable,
+                                       'variable',
+                                       )
+
+    def _convert_variable(self,
+                          variable,
+                          variable_type: str,
+                          ) -> None:
+        if variable.namespace == self.objectspace.rougailconfig['variable_namespace'] and \
+                variable.name in self.forbidden_name:
+            msg = _(f'the name of the variable "{variable.name}" cannot be the same as the name'
+                    ' of a namespace')
+            raise DictConsistencyError(msg, 54, variable.xmlfiles)
+        if variable.type != 'symlink' and not hasattr(variable, 'description'):
+            variable.description = variable.name
+        if hasattr(variable, 'value'):
+            value_to_del = []
+            for idx, value in enumerate(variable.value):
+                if not hasattr(value, 'name'):
+                    value_to_del.append(idx)
+                else:
+                    if not hasattr(value, 'type'):
+                        value.type = variable.type
+                    value.name = CONVERT_OPTION.get(value.type, {}).get('func', str)(value.name)
+            value_to_del.sort(reverse=True)
+            for idx in value_to_del:
+                del variable.value[idx]
+            if not variable.value:
+                del variable.value
+        variable.doc = variable.description
+        del variable.description
+        if variable_type == 'follower':
+            if variable.multi is True:
+                variable.multi = 'submulti'
+            else:
+                variable.multi = True
+        self._convert_valid_enum(variable)
+
+    def _convert_valid_enum(self,
+                            variable,
+                            ):
+        """some types are, in fact, choices
+        convert this kind of variables into choice
+        """
+        if variable.type in FORCE_CHOICE:
+            if not hasattr(self.objectspace.space, 'constraints'):
+                xmlfiles = variable.xmlfiles
+                self.objectspace.space.constraints = self.objectspace.constraints(xmlfiles)
+                self.objectspace.space.constraints.namespace = variable.namespace
+            if not hasattr(self.objectspace.space.constraints, 'check'):
+                self.objectspace.space.constraints.check = []
+            check = self.objectspace.check(variable.xmlfiles)
+            check.name = 'valid_enum'
+            target = self.objectspace.target(variable.xmlfiles)
+            target.name = variable.path
+            check.target = [target]
+            check.namespace = variable.namespace
+            check.param = []
+            for value in FORCE_CHOICE[variable.type]:
+                param = self.objectspace.param(variable.xmlfiles)
+                param.text = value
+                check.param.append(param)
+            self.objectspace.space.constraints.check.append(check)
+            variable.type = 'string'
+
+    def convert_test(self):
+        """Convert variable tests value
+        """
+        for variable in self.get_variables():
+            if not hasattr(variable, 'test'):
+                continue
+            if variable.test:
+                if not hasattr(variable, 'information'):
+                    variable.information = self.objectspace.information(variable.xmlfiles)
+                values = variable.test.split('|')
+                new_values = []
+                for value in values:
+                    if value == '':
+                        value = None
+                    else:
+                        value = CONVERT_OPTION.get(variable.type, {}).get('func', str)(value)
+                    new_values.append(value)
+                variable.information.test = tuple(new_values)
+            del variable.test
+
+    def convert_help(self):
+        """Convert variable help
+        """
+        for variable in self.get_variables():
+            if not hasattr(variable, 'information'):
+                variable.information = self.objectspace.information(variable.xmlfiles)
+            self._convert_help(variable)
+
+    @staticmethod
+    def _convert_help(variable) -> None:
+        if hasattr(variable, 'help'):
+            variable.information.help = variable.help
+            del variable.help

src/rougail/config.py

@@ -1,22 +1,50 @@
-# -*- coding: utf-8 -*-
 """
-fichier de configuration pour rougail
+Config file for Rougail
 
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 """
 from os.path import join, abspath, dirname
 
 
-rougailroot = '/var/rougail'
+ROUGAILROOT = '/srv/rougail'
-dtddir = join(dirname(abspath(__file__)), 'data')
+DTDDIR = join(dirname(abspath(__file__)), 'data')
 
 
-Config = {'rougailroot': rougailroot,
+RougailConfig = {'dictionaries_dir': [join(ROUGAILROOT, 'dictionaries')],
-          'patch_dir': join(rougailroot, 'patches'),
+                 'extra_dictionaries': {},
-          'manifests_dir': join(rougailroot, 'manifests'),
+                 'patches_dir': join(ROUGAILROOT, 'patches'),
-          'templates_dir': join(rougailroot, 'templates'),
+                 'templates_dir': join(ROUGAILROOT, 'templates'),
-          'dtdfilename': join(dtddir, 'rougail.dtd'),
+                 'destinations_dir': join(ROUGAILROOT, 'destinations'),
-          'dtddir': dtddir,
+                 'tmp_dir': join(ROUGAILROOT, 'tmp'),
-          # chemin du répertoire source des fichiers templates
+                 'dtdfilename': join(DTDDIR, 'rougail.dtd'),
-          'patch_dir': '/srv/rougail/patch',
+                 'functions_file': join(ROUGAILROOT, 'functions.py'),
                  'variable_namespace': 'rougail',
+                 'auto_freeze_variable': 'server_deployed',
+                 'default_engine': 'creole',
+                 'internal_functions': [],
+                 'modes_level': ['basic', 'normal', 'expert'],
+                 'default_family_mode': 'basic',
+                 'default_variable_mode': 'normal',
                  }

src/rougail/convert.py

@@ -0,0 +1,109 @@
+"""Takes a bunch of Rougail XML dispatched in differents folders
+as an input and outputs a Tiramisu's file.
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+Sample usage::
+
+    >>> from rougail import RougailConvert
+    >>> rougail = RougailConvert()
+    >>> tiramisu = rougail.save('tiramisu.py')
+
+The Rougail
+
+- loads the XML into an internal RougailObjSpace representation
+- visits/annotates the objects
+- dumps the object space as Tiramisu string
+
+The visit/annotation stage is a complex step that corresponds to the Rougail
+procedures.
+"""
+
+from typing import List
+
+from .i18n import _
+from .config import RougailConfig
+from .objspace import RougailObjSpace
+from .xmlreflector import XMLReflector
+from .tiramisureflector import TiramisuReflector
+from .annotator import SpaceAnnotator
+from .error import DictConsistencyError
+
+
+class RougailConvert:
+    """Rougail object
+    """
+    def __init__(self,
+                 rougailconfig: RougailConfig=None,
+                 ) -> None:
+        if rougailconfig is None:
+            rougailconfig = RougailConfig
+        xmlreflector = XMLReflector(rougailconfig)
+        rougailobjspace = RougailObjSpace(xmlreflector,
+                                          rougailconfig,
+                                          )
+        self._load_dictionaries(xmlreflector,
+                                rougailobjspace,
+                                rougailconfig['variable_namespace'],
+                                rougailconfig['dictionaries_dir'],
+                                )
+        for namespace, extra_dir in rougailconfig['extra_dictionaries'].items():
+            if namespace in ['services', rougailconfig['variable_namespace']]:
+                msg = _(f'Namespace name "{namespace}" is not allowed')
+                raise DictConsistencyError(msg, 21, None)
+            self._load_dictionaries(xmlreflector,
+                                    rougailobjspace,
+                                    namespace,
+                                    extra_dir,
+                                    )
+        functions_file = rougailconfig['functions_file']
+        SpaceAnnotator(rougailobjspace,
+                       functions_file,
+                       )
+        self.output = TiramisuReflector(rougailobjspace,
+                                        functions_file,
+                                        ).get_text() + '\n'
+
+    @staticmethod
+    def _load_dictionaries(xmlreflector: XMLReflector,
+                           rougailobjspace: RougailObjSpace,
+                           namespace: str,
+                           xmlfolders: List[str],
+                           ) -> List[str]:
+        for xmlfile, document in xmlreflector.load_xml_from_folders(xmlfolders):
+            rougailobjspace.xml_parse_document(xmlfile,
+                                               document,
+                                               namespace,
+                                               )
+
+    def save(self,
+             filename: str,
+             ) -> str:
+        """Return tiramisu object declaration as a string
+        """
+        if filename:
+            with open(filename, 'w') as tiramisu:
+                tiramisu.write(self.output)
+        return self.output

src/rougail/data/rougail.dtd

@@ -13,7 +13,7 @@
 
 #    Forked by:
 #    Cadoles (http://www.cadoles.com)
-#    Copyright (C) 2019-2020
+#    Copyright (C) 2019-2021
 
 #    distribued with GPL-2 or later license
 
@@ -37,7 +37,8 @@
 <!-- root element    -->
 <!-- =============== -->
 
-<!ELEMENT rougail (services | variables | constraints | help)*>
+<!ELEMENT rougail (services|variables|constraints)*>
+<!ATTLIST rougail version (0.9) #REQUIRED>
 
 <!-- ============== -->
 <!-- files element  -->
@@ -45,52 +46,49 @@
 
 <!ELEMENT services (service*)>
 
-<!ELEMENT service ((port* | tcpwrapper* | ip* | interface* | package* | file* | override*)*) >
+<!ELEMENT service ((ip*|file*|override*)*)>
 <!ATTLIST service name CDATA #REQUIRED>
 <!ATTLIST service manage (True|False) "True">
-
+<!ATTLIST service servicelist CDATA #IMPLIED>
-<!ELEMENT port (#PCDATA)>
-<!ATTLIST port port_type (PortOption|SymLinkOption|variable) "PortOption">
-<!ATTLIST port portlist CDATA #IMPLIED >
-<!ATTLIST port protocol (tcp|udp) "tcp">
 
 <!ELEMENT ip (#PCDATA)>
-<!ATTLIST ip iplist CDATA #IMPLIED >
+<!ATTLIST ip iplist CDATA #IMPLIED>
-<!ATTLIST ip ip_type (NetworkOption|SymLinkOption|variable) "NetworkOption">
+<!ATTLIST ip ip_type (variable) "variable">
-<!ATTLIST ip interface_type (UnicodeOption|SymLinkOption|variable) "UnicodeOption">
+<!ATTLIST ip netmask_type (variable) "variable">
-<!ATTLIST ip interface CDATA #REQUIRED>
+<!ATTLIST ip netmask CDATA #IMPLIED>
-<!ATTLIST ip netmask_type (NetmaskOption|SymLinkOption|variable) "NetmaskOption">
-<!ATTLIST ip netmask CDATA "255.255.255.255">
 
-<!ELEMENT file EMPTY>
+<!ELEMENT file (#PCDATA)>
-<!ATTLIST file name CDATA #REQUIRED >
+<!ATTLIST file file_type (filename|variable) "filename">
-<!ATTLIST file file_type (UnicodeOption|SymLinkOption|variable) "UnicodeOption">
 <!ATTLIST file variable CDATA #IMPLIED>
 <!ATTLIST file variable_type (variable) "variable">
 <!ATTLIST file source CDATA #IMPLIED>
 <!ATTLIST file mode CDATA "0644">
 <!ATTLIST file owner CDATA "root">
 <!ATTLIST file group CDATA "root">
-<!ATTLIST file filelist CDATA #IMPLIED >
+<!ATTLIST file filelist CDATA #IMPLIED>
 <!ATTLIST file redefine (True|False) "False">
-<!ATTLIST file templating (True|False) "True">
+<!ATTLIST file engine (none|creole|jinja2|creole_legacy) #IMPLIED>
+<!ATTLIST file included (no|name|content) "no">
 
 <!ELEMENT override EMPTY>
-<!ATTLIST override source CDATA #IMPLIED >
+<!ATTLIST override source CDATA #IMPLIED>
-<!ATTLIST override templating (True|False) "True">
+<!ATTLIST override engine (none|creole|jinja2) #IMPLIED>
 
-<!ELEMENT variables (family*, separators*)>
+<!ELEMENT variables ((variable*|family*)*)>
-<!ELEMENT family (#PCDATA | variable)*>
+
+<!ELEMENT family ((variable*|family*)*)>
 <!ATTLIST family name CDATA #REQUIRED>
 <!ATTLIST family description CDATA #IMPLIED>
-<!ATTLIST family mode (basic|normal|expert) "basic">
+<!ATTLIST family help CDATA #IMPLIED>
+<!ATTLIST family mode CDATA #IMPLIED>
 <!ATTLIST family hidden (True|False) "False">
 <!ATTLIST family dynamic CDATA #IMPLIED>
 
-<!ELEMENT variable (#PCDATA | value)*>
+<!ELEMENT variable (value*)>
 <!ATTLIST variable name CDATA #REQUIRED>
-<!ATTLIST variable type CDATA #IMPLIED>
+<!ATTLIST variable type (number|float|string|password|mail|boolean|filename|date|unix_user|ip|local_ip|netmask|network|broadcast|netbios|domainname|hostname|web_address|port|mac|cidr|network_cidr|schedule|schedulemod) "string">
 <!ATTLIST variable description CDATA #IMPLIED>
+<!ATTLIST variable help CDATA #IMPLIED>
 <!ATTLIST variable hidden (True|False) "False">
 <!ATTLIST variable disabled (True|False) "False">
 <!ATTLIST variable multi (True|False) "False">
@@ -99,51 +97,42 @@
 <!ATTLIST variable mandatory (True|False) "False">
 <!ATTLIST variable auto_freeze (True|False) "False">
 <!ATTLIST variable auto_save (True|False) "False">
-<!ATTLIST variable mode (basic|normal|expert) "normal">
+<!ATTLIST variable mode CDATA #IMPLIED>
 <!ATTLIST variable remove_check (True|False) "False">
 <!ATTLIST variable remove_condition (True|False) "False">
 <!ATTLIST variable remove_fill (True|False) "False">
 <!ATTLIST variable test CDATA #IMPLIED>
 
-<!ELEMENT separators (separator*)>
-
-<!ELEMENT separator (#PCDATA)>
-<!ATTLIST separator name CDATA #REQUIRED>
-
 <!ELEMENT value (#PCDATA)>
 
-<!ELEMENT constraints ((fill* | check* | condition* | group*)*)>
+<!ELEMENT constraints ((fill*|check*|condition*|group*)*)>
-<!ELEMENT fill (param*)>
-<!ATTLIST fill name CDATA #REQUIRED>
-<!ATTLIST fill target CDATA #REQUIRED>
 
-<!ELEMENT check (param*)>
+<!ELEMENT fill ((target|param)+)>
+<!ATTLIST fill name CDATA #REQUIRED>
+
+<!ELEMENT check ((target|param)+)>
 <!ATTLIST check name CDATA #REQUIRED>
-<!ATTLIST check target CDATA #REQUIRED>
 <!ATTLIST check level (error|warning) "error">
 
-<!ELEMENT condition ((target | param)+ )>
+<!ELEMENT condition ((target|param)+)>
-<!ATTLIST condition name (disabled_if_in|disabled_if_not_in|hidden_if_in|auto_hidden_if_not_in|hidden_if_not_in|mandatory_if_in|mandatory_if_not_in) #REQUIRED>
+<!ATTLIST condition name (disabled_if_in|disabled_if_not_in|hidden_if_in|hidden_if_not_in|mandatory_if_in|mandatory_if_not_in) #REQUIRED>
 <!ATTLIST condition source CDATA #REQUIRED>
-<!ATTLIST condition fallback (True|False) "False">
+<!ATTLIST condition optional (True|False) "False">
-<!ATTLIST condition force_condition_on_fallback (True|False) "False">
+<!ATTLIST condition apply_on_fallback (True|False) #IMPLIED>
-<!ATTLIST condition force_inverse_condition_on_fallback (True|False) "False">
+
+<!ELEMENT param (#PCDATA)>
+<!ATTLIST param type (string|number|nil|boolean|variable|function|information|target_information|suffix|index) "string">
+<!ATTLIST param name CDATA #IMPLIED>
+<!ATTLIST param propertyerror (True|False) "True">
+<!ATTLIST param optional (True|False) "False">
+
+<!ELEMENT target (#PCDATA)>
+<!ATTLIST target type (variable|family|servicelist|filelist|iplist) "variable">
+<!ATTLIST target optional (True|False) "False">
 
 <!ELEMENT group (follower+)>
 <!ATTLIST group leader CDATA #REQUIRED>
 <!ATTLIST group name CDATA #IMPLIED>
 <!ATTLIST group description CDATA #IMPLIED>
 
-<!ELEMENT param (#PCDATA)>
-<!ATTLIST param type (string|number|variable|information|suffix) "string">
-<!ATTLIST param name CDATA #IMPLIED>
-<!ATTLIST param notraisepropertyerror (True|False) "False">
-<!ATTLIST param optional (True|False) "False">
-
-<!ELEMENT target (#PCDATA)>
-<!ATTLIST target type (family|variable|filelist|iplist|portlist) "variable">
-<!ATTLIST target optional (True|False) "False">
-
 <!ELEMENT follower (#PCDATA)>
-
-<!ELEMENT help ((variable* | family*)*)>

src/rougail/error.py

@@ -1,22 +1,58 @@
-# -*- coding: utf-8 -*-
+"""Standard error classes
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from .i18n import _
+
+
+def display_xmlfiles(xmlfiles: list) -> str:
+    """The function format xmlfiles informations to generate errors
+    """
+    if len(xmlfiles) == 1:
+        return '"' + xmlfiles[0] + '"'
+    return '"' + '", "'.join(xmlfiles[:-1]) + '"' + ' and ' + '"' + xmlfiles[-1] + '"'
+
+
 class ConfigError(Exception):
-    pass
+    """Standard error for templating
+    """
+
 
 class FileNotFound(ConfigError):
-    pass
+    """Template file is not found
+    """
+
 
 class TemplateError(ConfigError):
-    pass
+    """Templating generate an error
+    """
+
 
 class TemplateDisabled(TemplateError):
     """Template is disabled.
     """
-    pass
-
-
-class OperationError(Exception):
-    """Type error or value Error for Creole variable's type or values
-    """
 
 
 class SpaceObjShallNotBeUpdated(Exception):
@@ -29,7 +65,13 @@ class DictConsistencyError(Exception):
     """It's not only that the Creole XML is valid against the Creole DTD
     it's that it is not consistent.
     """
+    def __init__(self, msg, errno, xmlfiles):
+        if xmlfiles:
+            msg = _(f'{msg} in {display_xmlfiles(xmlfiles)}')
+        super().__init__(msg)
+        self.errno = errno
 
 
-class LoaderError(Exception):
+class UpgradeError(Exception):
-    pass
+    """Error during XML upgrade
+    """

src/rougail/i18n.py

@@ -1,23 +1,28 @@
-# -*- coding: UTF-8 -*-
+"""Internationalisation utilities
-# Copyright (C) 2012-2013 Team tiramisu (see AUTHORS for all contributors)
+Created by:
-#
+EOLE (http://eole.orion.education.fr)
-# This program is free software: you can redistribute it and/or modify it
+Copyright (C) 2005-2018
-# under the terms of the GNU Lesser General Public License as published by the
+
-# Free Software Foundation, either version 3 of the License, or (at your
+Forked by:
-# option) any later version.
+Cadoles (http://www.cadoles.com)
-#
+Copyright (C) 2019-2021
-# This program is distributed in the hope that it will be useful, but WITHOUT
+
-# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+distribued with GPL-2 or later license
-# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
+
-# details.
+This program is free software; you can redistribute it and/or modify
-#
+it under the terms of the GNU General Public License as published by
-# You should have received a copy of the GNU Lesser General Public License
+the Free Software Foundation; either version 2 of the License, or
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+(at your option) any later version.
-#
+
-# The original `Config` design model is unproudly borrowed from
+This program is distributed in the hope that it will be useful,
-# the rough gus of pypy: pypy: http://codespeak.net/svn/pypy/dist/pypy/config/
+but WITHOUT ANY WARRANTY; without even the implied warranty of
-# the whole pypy projet is under MIT licence
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-"internationalisation utilities"
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
 import gettext
 import os
 import sys

src/rougail/objspace.py

@@ -1,514 +1,413 @@
+"""parse XML files and build a space with objects
+it aggregates this files and manage redefine and exists attributes
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 """
-Creole flattener. Takes a bunch of Creole XML dispatched in differents folders
+from typing import Optional
-as an input and outputs a human readable flatened XML
-
-Sample usage::
-
-    >>> from rougail.objspace import CreoleObjSpace
-    >>> eolobj = CreoleObjSpace('/usr/share/rougail/rougail.dtd')
-    >>> eolobj.create_or_populate_from_xml('rougail', ['/usr/share/eole/rougail/dicos'])
-    >>> eolobj.space_visitor()
-    >>> eolobj.save('/tmp/rougail_flatened_output.xml')
-
-The CreoleObjSpace
-
-- loads the XML into an internal CreoleObjSpace representation
-- visits/annotates the objects
-- dumps the object space as XML output into a single XML target
-
-The visit/annotation stage is a complex step that corresponds to the Creole
-procedures.
-
-For example: a variable is redefined and shall be moved to another family
-means that a variable1 = Variable() object in the object space who lives in the family1 parent
-has to be moved in family2. The visit procedure changes the varable1's object space's parent.
-"""
-from lxml.etree import Element, SubElement  # pylint: disable=E0611
 
 from .i18n import _
 from .xmlreflector import XMLReflector
-from .annotator import ERASED_ATTRIBUTES, SpaceAnnotator
+from .utils import valid_variable_family_name
-from .tiramisureflector import TiramisuReflector
+from .error import SpaceObjShallNotBeUpdated, DictConsistencyError
-from .utils import normalize_family
-from .error import OperationError, SpaceObjShallNotBeUpdated, DictConsistencyError
 from .path import Path
-from .config import Config
 
-# CreoleObjSpace's elements like 'family' or 'follower', that shall be forced to the Redefinable type
+# RougailObjSpace's elements that shall be forced to the Redefinable type
 FORCE_REDEFINABLES = ('family', 'follower', 'service', 'disknod', 'variables')
-# CreoleObjSpace's elements that shall be forced to the UnRedefinable type
+# RougailObjSpace's elements that shall be forced to the UnRedefinable type
 FORCE_UNREDEFINABLES = ('value',)
-# CreoleObjSpace's elements that shall be set to the UnRedefinable type
+# RougailObjSpace's elements that shall not be modify
 UNREDEFINABLE = ('multi', 'type')
+# RougailObjSpace's elements that did not created automaticly
+FORCE_ELEMENTS = ('choice', 'property_', 'leadership', 'information')
+# XML text are convert has name
+FORCED_TEXT_ELTS_AS_NAME = ('choice', 'property', 'value',)
 
-CONVERT_PROPERTIES = {'auto_save': ['force_store_value'], 'auto_freeze': ['force_store_value', 'auto_freeze']}
+FORCE_TAG = {'family': 'variable'}
 
-RENAME_ATTIBUTES = {'description': 'doc'}
-
-FORCED_TEXT_ELTS_AS_NAME = ('choice', 'property', 'value', 'target')
-
-CONVERT_EXPORT = {'Leadership': 'leader',
-                  'Variable': 'variable',
-                  'Value': 'value',
-                  'Property': 'property',
-                  'Choice': 'choice',
-                  'Param': 'param',
-                  'Check': 'check',
-                  }
 
 # _____________________________________________________________________________
 # special types definitions for the Object Space's internal representation
-class RootCreoleObject:
+class RootRougailObject:  # pylint: disable=R0903
-    def __init__(self, xmlfiles):
+    """Root object
+    """
+    def __init__(self,
+                 xmlfiles,
+                 name=None,
+                 ):
         if not isinstance(xmlfiles, list):
             xmlfiles = [xmlfiles]
         self.xmlfiles = xmlfiles
+        if name:
+            self.name = name
 
 
-class CreoleObjSpace:
+class Atom(RootRougailObject):  # pylint: disable=R0903
-    """DOM XML reflexion free internal representation of a Creole Dictionary
+    """Atomic object (means can only define one time)
     """
-    choice = type('Choice', (RootCreoleObject,), dict())
-    property_ = type('Property', (RootCreoleObject,), dict())
-    # Creole ObjectSpace's Leadership variable class type
-    Leadership = type('Leadership', (RootCreoleObject,), dict())
-    """
-    This Atom type stands for singleton, that is
-    an Object Space's atom object is present only once in the
-    object space's tree
-    """
-    Atom = type('Atom', (RootCreoleObject,), dict())
-    "A variable that can't be redefined"
-    Redefinable = type('Redefinable', (RootCreoleObject,), dict())
-    "A variable can be redefined"
-    UnRedefinable = type('UnRedefinable', (RootCreoleObject,), dict())
 
 
-    def __init__(self, dtdfilename):  # pylint: disable=R0912
+class Redefinable(RootRougailObject):  # pylint: disable=R0903
-        self.index = 0
+    """Object that could be redefine
-        class ObjSpace:  # pylint: disable=R0903
+    """
+
+
+class UnRedefinable(RootRougailObject):  # pylint: disable=R0903
+    """Object that could not be redefine
+    """
+
+
+class ObjSpace:  # pylint: disable=R0903
     """
     Base object space
     """
-        self.space = ObjSpace()
-        self.paths = Path()
-        self.xmlreflector = XMLReflector()
-        self.xmlreflector.parse_dtd(dtdfilename)
-        self.redefine_variables = None
-        self.fill_removed = None
-        self.check_removed = None
-        self.condition_removed = None
 
-        # ['variable', 'separator', 'family']
+
-        self.forced_text_elts = set()
+def convert_boolean(value: str) -> bool:
+    """Boolean coercion. The Rougail XML may contain srings like `True` or `False`
+    """
+    if isinstance(value, bool):
+        return value
+    if value == 'True':
+        return True
+    return False
+
+
+class RougailObjSpace:
+    """Rougail ObjectSpace is an object's reflexion of the XML elements
+    """
+
+    def __init__(self,
+                 xmlreflector: XMLReflector,
+                 rougailconfig: 'RougailConfig',
+                 ) -> None:
+        self.space = ObjSpace()
+        self.paths = Path(rougailconfig)
+
         self.forced_text_elts_as_name = set(FORCED_TEXT_ELTS_AS_NAME)
         self.list_conditions = {}
+        self.valid_enums = {}
         self.booleans_attributs = []
+        self.has_dyn_option = False
+        self.types = {}
 
-        self.make_object_space_class()
+        self.make_object_space_classes(xmlreflector)
+        self.rougailconfig = rougailconfig
 
-    def make_object_space_class(self):
+    def make_object_space_classes(self,
-        """Create Rougail ObjectSpace class types, it enables us to create objects like:
+                                  xmlreflector: XMLReflector,
+                                  ) -> None:
+        """Create Rougail ObjectSpace class types from DDT file
+        It enables us to create objects like:
         File(), Variable(), Ip(), Family(), Constraints()... and so on.
+        """
 
-        Creole ObjectSpace is an object's reflexion of the XML elements"""
+        for dtd_elt in xmlreflector.dtd.iterelements():
-
-        for dtd_elt in self.xmlreflector.dtd.iterelements():
             attrs = {}
             if dtd_elt.name in FORCE_REDEFINABLES:
-                clstype = self.Redefinable
+                clstype = Redefinable
+            elif not dtd_elt.attributes() and dtd_elt.name not in FORCE_UNREDEFINABLES:
+                clstype = Atom
             else:
-                clstype = self.UnRedefinable
+                clstype = UnRedefinable
-            atomic = dtd_elt.name not in FORCE_UNREDEFINABLES and dtd_elt.name not in FORCE_REDEFINABLES
             forced_text_elt = dtd_elt.type == 'mixed'
             for dtd_attr in dtd_elt.iterattributes():
-                atomic = False
+                if set(dtd_attr.itervalues()) == {'True', 'False'}:
-                if set(dtd_attr.itervalues()) == set(['True', 'False']):
                     # it's a boolean
                     self.booleans_attributs.append(dtd_attr.name)
                 if dtd_attr.default_value:
                     # set default value for this attribute
                     default_value = dtd_attr.default_value
                     if dtd_attr.name in self.booleans_attributs:
-                        default_value = self.convert_boolean(dtd_attr.default_value)
+                        default_value = convert_boolean(default_value)
                     attrs[dtd_attr.name] = default_value
+                    if dtd_attr.name.endswith('_type'):
+                        self.types[dtd_attr.name] = default_value
                 if dtd_attr.name == 'redefine':
                     # has a redefine attribute, so it's a Redefinable object
-                    clstype = self.Redefinable
+                    clstype = Redefinable
                 if dtd_attr.name == 'name' and forced_text_elt:
                     # child.text should be transform has a "name" attribute
-                    self.forced_text_elts.add(dtd_elt.name)
                     forced_text_elt = False
 
             if forced_text_elt is True:
                 self.forced_text_elts_as_name.add(dtd_elt.name)
-            if atomic:
-                # has any attribute so it's an Atomic object
-                clstype = self.Atom
 
             # create ObjectSpace object
             setattr(self, dtd_elt.name, type(dtd_elt.name.capitalize(), (clstype,), attrs))
+        for elt in FORCE_ELEMENTS:
+            setattr(self, elt, type(self._get_elt_name(elt), (RootRougailObject,), dict()))
 
-    def create_or_populate_from_xml(self,
+    @staticmethod
-                                    namespace,
+    def _get_elt_name(elt) -> str:
-                                    xmlfolders):
+        name = elt.capitalize()
-        """Parses a bunch of XML files
+        if name.endswith('_'):
-        populates the CreoleObjSpace
+            name = name[:-1]
-        """
+        return name
-        for xmlfile, document in self.xmlreflector.load_xml_from_folders(xmlfolders):
-            self.redefine_variables = []
-            self.fill_removed = []
-            self.check_removed = []
-            self.condition_removed = []
-            self.xml_parse_document(xmlfile,
-                                    document,
-                                    self.space,
-                                    namespace,
-                                    )
 
     def xml_parse_document(self,
                            xmlfile,
                            document,
-                           space,
                            namespace,
                            ):
-        """Parses a Creole XML file
+        """Parses a Rougail XML file and populates the RougailObjSpace
-        populates the CreoleObjSpace
         """
+        redefine_variables = []
+        self._xml_parse(xmlfile,
+                        document,
+                        self.space,
+                        namespace,
+                        redefine_variables,
+                        )
+
+    def _xml_parse(self,  # pylint: disable=R0913
+                   xmlfile,
+                   document,
+                   space,
+                   namespace,
+                   redefine_variables,
+                   ) -> None:
+        # var to check unique family name in a XML file
         family_names = []
         for child in document:
-            # this index enables us to reorder objects
-            self.index += 1
-            # doesn't proceed the XML commentaries
             if not isinstance(child.tag, str):
+                # doesn't proceed the XML commentaries
                 continue
             if child.tag == 'family':
                 if child.attrib['name'] in family_names:
-                    raise DictConsistencyError(_(f'Family "{child.attrib["name"]}" is set several times in "{xmlfile}"'))
+                    msg = _(f'Family "{child.attrib["name"]}" is set several times')
+                    raise DictConsistencyError(msg, 44, [xmlfile])
                 family_names.append(child.attrib['name'])
-            if child.tag == 'variables':
-                child.attrib['name'] = namespace
-            if child.tag == 'value' and child.text == None:
-                # FIXME should not be here
-                continue
-            # variable objects creation
             try:
-                variableobj = self.generate_variableobj(xmlfile,
+                # variable objects creation
+                exists, variableobj = self.get_variableobj(xmlfile,
                                                            child,
                                                            space,
                                                            namespace,
+                                                           redefine_variables,
                                                            )
             except SpaceObjShallNotBeUpdated:
                 continue
-            self.set_text_to_obj(child,
+            self.set_text(child,
                           variableobj,
                           )
-            self.set_xml_attributes_to_obj(xmlfile,
+            self.set_attributes(xmlfile,
                                 child,
                                 variableobj,
                                 )
-            self.variableobj_tree_visitor(child,
+            self.remove(child,
                         variableobj,
-                                          namespace,
+                        redefine_variables,
                         )
-            self.fill_variableobj_path_attribute(space,
+            if not exists:
-                                                 child,
+                self.set_path(namespace,
-                                                 namespace,
                               document,
                               variableobj,
+                              space,
                               )
             self.add_to_tree_structure(variableobj,
                                        space,
                                        child,
+                                       namespace,
                                        )
             if list(child) != []:
-                self.xml_parse_document(xmlfile,
+                self._xml_parse(xmlfile,
                                 child,
                                 variableobj,
                                 namespace,
+                                redefine_variables,
                                 )
 
-    def generate_variableobj(self,
+    def get_variableobj(self,
-                             xmlfile,
+                        xmlfile: str,
-                             child,
+                        child: list,
                         space,
                         namespace,
-                             ):
+                        redefine_variables,
+                        ):  # pylint: disable=R0913
         """
-        instanciates or creates Creole Object Subspace objects
+        retrieves or creates Rougail Object Subspace objects
         """
-        variableobj = getattr(self, child.tag)(xmlfile)
+        tag = FORCE_TAG.get(child.tag, child.tag)
-        if isinstance(variableobj, self.Redefinable):
+        obj = getattr(self, tag)
-            variableobj = self.create_or_update_redefinable_object(xmlfile,
+        name = self._get_name(child, namespace)
+        if Redefinable in obj.__mro__:
+            return self.create_or_update_redefinable_object(xmlfile,
                                                             child.attrib,
                                                             space,
                                                             child,
+                                                            name,
                                                             namespace,
+                                                            redefine_variables,
                                                             )
-        elif isinstance(variableobj, self.Atom) and child.tag in vars(space):
+        if Atom in obj.__mro__:
-            # instanciates an object from the CreoleObjSpace's builtins types
+            if child.tag in vars(space):
-            # example : child.tag = constraints -> a self.Constraints() object is created
+                # Atom instance has to be a singleton here
-            # this Atom instance has to be a singleton here
                 # we do not re-create it, we reuse it
-            variableobj = getattr(space, child.tag)
+                return False, getattr(space, child.tag)
-        self.create_tree_structure(space,
+            return False, obj(xmlfile, name)
+        # UnRedefinable object
+        if child.tag not in vars(space):
+            setattr(space, child.tag, [])
+        return False, obj(xmlfile, name)
+
+    def _get_name(self,
                   child,
-                                   variableobj,
+                  namespace: str,
-                                   )
+                  ) -> Optional[str]:
-        return variableobj
+        if child.tag == 'variables':
+            return namespace
+        if 'name' in child.attrib:
+            return child.attrib['name']
+        if child.text and child.tag in self.forced_text_elts_as_name:
+            return child.text.strip()
+        return None
 
     def create_or_update_redefinable_object(self,
                                             xmlfile,
                                             subspace,
                                             space,
                                             child,
+                                            name,
                                             namespace,
-                                            ):
+                                            redefine_variables,
-        """Creates or retrieves the space object that corresponds
+                                            ):  # pylint: disable=R0913
-        to the `child` XML object
+        """A redefinable object could be created or updated
-
-        Two attributes of the `child` XML object are important:
-
-        - with the `redefine` boolean flag attribute we know whether
-          the corresponding space object shall be created or updated
-
-            - `True` means that the corresponding space object shall be updated
-            - `False` means that the corresponding space object shall be created
-
-        - with the `exists` boolean flag attribute we know whether
-          the corresponding space object shall be created
-          (or nothing -- that is the space object isn't modified)
-
-            - `True` means that the corresponding space object shall be created
-            - `False` means that the corresponding space object is not updated
-
-        In the special case `redefine` is True and `exists` is False,
-        we create the corresponding space object if it doesn't exist
-        and we update it if it exists.
-
-        :return: the corresponding space object of the `child` XML object
         """
-        if child.tag in self.forced_text_elts_as_name:
+        existed_var = self.get_existed_obj(name,
-            name = child.text
+                                           xmlfile,
-        else:
-            name = subspace['name']
-        existed_var = self.is_already_exists(name,
                                            space,
                                            child,
                                            namespace,
                                            )
         if existed_var:
+            # if redefine is set to object, default value is False
+            # otherwise it's always a redefinable object
             default_redefine = child.tag in FORCE_REDEFINABLES
-            redefine = self.convert_boolean(subspace.get('redefine', default_redefine))
+            redefine = convert_boolean(subspace.get('redefine', default_redefine))
-            exists = self.convert_boolean(subspace.get('exists', True))
             if redefine is True:
-                existed_var.xmlfiles.append(xmlfile)
+                if isinstance(existed_var, self.variable):  # pylint: disable=E1101
-                return self.translate_in_space(name,
+                    if namespace == self.rougailconfig['variable_namespace']:
-                                               space,
+                        redefine_variables.append(name)
-                                               child,
-                                               namespace,
-                                               )
-            elif exists is False:
-                raise SpaceObjShallNotBeUpdated()
-            xmlfiles = self.display_xmlfiles(existed_var.xmlfiles)
-            raise DictConsistencyError(_(f'"{child.tag}" named "{name}" cannot be re-created in "{xmlfile}", already defined in {xmlfiles}'))
-        redefine = self.convert_boolean(subspace.get('redefine', False))
-        exists = self.convert_boolean(subspace.get('exists', False))
-        if redefine is False or exists is True:
-            return getattr(self, child.tag)(xmlfile)
-        raise DictConsistencyError(_(f'Redefined object in "{xmlfile}": "{name}" does not exist yet'))
-
-    def display_xmlfiles(self,
-                         xmlfiles: list,
-                         ) -> str:
-        if len(xmlfiles) == 1:
-            return '"' + xmlfiles[0] + '"'
                     else:
-            return '"' + '", "'.join(xmlfiles[:-1]) + '"' + ' and ' + '"' + xmlfiles[-1] + '"'
+                        redefine_variables.append(space.path + '.' + name)
+                existed_var.xmlfiles.append(xmlfile)
+                return True, existed_var
+            exists = convert_boolean(subspace.get('exists', True))
+            if exists is False:
+                raise SpaceObjShallNotBeUpdated()
+            msg = _(f'"{child.tag}" named "{name}" cannot be re-created in "{xmlfile}", '
+                    f'already defined')
+            raise DictConsistencyError(msg, 45, existed_var.xmlfiles)
+        # object deos not exists
+        exists = convert_boolean(subspace.get('exists', False))
+        if exists is True:
+            # manage object only if already exists, so cancel
+            raise SpaceObjShallNotBeUpdated()
+        redefine = convert_boolean(subspace.get('redefine', False))
+        if redefine is True:
+            # cannot redefine an inexistant object
+            msg = _(f'Redefined object: "{name}" does not exist yet')
+            raise DictConsistencyError(msg, 46, [xmlfile])
+        tag = FORCE_TAG.get(child.tag, child.tag)
+        if tag not in vars(space):
+            setattr(space, tag, {})
+        obj = getattr(self, child.tag)(xmlfile, name)
+        return False, obj
 
-    def create_tree_structure(self,
+    def get_existed_obj(self,
-                              space,
-                              child,
-                              variableobj,
-                              ):  # pylint: disable=R0201
-        """
-        Builds the tree structure of the object space here
-        we set services attributes in order to be populated later on
-        for example::
-
-            space = Family()
-            space.variable = dict()
-            another example:
-            space = Variable()
-            space.value = list()
-        """
-        if child.tag not in vars(space):
-            if isinstance(variableobj, self.Redefinable):
-                setattr(space, child.tag, dict())
-            elif isinstance(variableobj, self.UnRedefinable):
-                setattr(space, child.tag, [])
-            elif not isinstance(variableobj, self.Atom):  # pragma: no cover
-                raise OperationError(_("Creole object {} "
-                                             "has a wrong type").format(type(variableobj)))
-
-    def is_already_exists(self,
                         name: str,
+                        xmlfile: str,
                         space: str,
                         child,
                         namespace: str,
-                          ):
+                        ) -> None:
-        if isinstance(space, self.family):  # pylint: disable=E1101
+        """if an object exists, return it
-            if namespace != Config['variable_namespace']:
-                name = space.path + '.' + name
-            if self.paths.path_is_defined(name):
-                return self.paths.get_variable_obj(name)
-            return
-        if child.tag == 'family':
-            norm_name = normalize_family(name)
-        else:
-            norm_name = name
-        children = getattr(space, child.tag, {})
-        if norm_name in children:
-            return children[norm_name]
-
-    def convert_boolean(self, value):  # pylint: disable=R0201
-        """Boolean coercion. The Creole XML may contain srings like `True` or `False`
         """
-        if isinstance(value, bool):
+        if child.tag in ['variable', 'family']:
-            return value
+            valid_variable_family_name(name, [xmlfile])
-        if value == 'True':
+        if child.tag == 'variable':  # pylint: disable=E1101
-            return True
+            if namespace != self.rougailconfig['variable_namespace']:
-        elif value == 'False':
+                name = space.path + '.' + name
-            return False
+            if not self.paths.path_is_defined(name):
-        else:
+                return None
-            raise TypeError(_('{} is not True or False').format(value))  # pragma: no cover
+            old_family_name = self.paths.get_variable_family_path(name)
+            if space.path != old_family_name:
+                msg = _(f'Variable "{name}" was previously create in family "{old_family_name}", '
+                        f'now it is in "{space.path}"')
+                raise DictConsistencyError(msg, 47, space.xmlfiles)
+            return self.paths.get_variable(name)
+        # it's not a family
+        tag = FORCE_TAG.get(child.tag, child.tag)
+        children = getattr(space, tag, {})
+        if name in children and isinstance(children[name], getattr(self, child.tag)):
+            return children[name]
+        return None
 
-    def translate_in_space(self,
+    def set_text(self,
-                           name,
-                           family,
-                           variable,
-                           namespace,
-                           ):
-        if not isinstance(family, self.family):  # pylint: disable=E1101
-            if variable.tag == 'family':
-                norm_name = normalize_family(name)
-            else:
-                norm_name = name
-            return getattr(family, variable.tag)[norm_name]
-        if namespace == Config['variable_namespace']:
-            path = name
-        else:
-            path = family.path + '.' + name
-        old_family_name = self.paths.get_variable_family_name(path)
-        if normalize_family(family.name) == old_family_name:
-            return getattr(family, variable.tag)[name]
-        old_family = self.space.variables[Config['variable_namespace']].family[old_family_name]  # pylint: disable=E1101
-        variable_obj = old_family.variable[name]
-        del old_family.variable[name]
-        if 'variable' not in vars(family):
-            family.variable = dict()
-        family.variable[name] = variable_obj
-        self.paths.add_variable(namespace,
-                                name,
-                                family.name,
-                                False,
-                                variable_obj,
-                                )
-        return variable_obj
-
-    def remove_fill(self, name):  # pylint: disable=C0111
-        if hasattr(self.space, 'constraints') and hasattr(self.space.constraints, 'fill'):
-            remove_fills= []
-            for idx, fill in enumerate(self.space.constraints.fill):  # pylint: disable=E1101
-                if hasattr(fill, 'target') and fill.target == name:
-                    remove_fills.append(idx)
-
-            remove_fills = list(set(remove_fills))
-            remove_fills.sort(reverse=True)
-            for idx in remove_fills:
-                self.space.constraints.fill.pop(idx)  # pylint: disable=E1101
-
-    def remove_check(self, name):  # pylint: disable=C0111
-        if hasattr(self.space, 'constraints') and hasattr(self.space.constraints, 'check'):
-            remove_checks = []
-            for idx, check in enumerate(self.space.constraints.check):  # pylint: disable=E1101
-                if hasattr(check, 'target') and check.target == name:
-                    remove_checks.append(idx)
-
-            remove_checks = list(set(remove_checks))
-            remove_checks.sort(reverse=True)
-            for idx in remove_checks:
-                self.space.constraints.check.pop(idx)  # pylint: disable=E1101
-
-    def remove_condition(self, name):  # pylint: disable=C0111
-        for idx, condition in enumerate(self.space.constraints.condition):  # pylint: disable=E1101
-            remove_targets = []
-            if hasattr(condition, 'target'):
-                for target_idx, target in enumerate(condition.target):
-                    if target.name == name:
-                        remove_targets.append(target_idx)
-            remove_targets = list(set(remove_targets))
-            remove_targets.sort(reverse=True)
-            for idx in remove_targets:
-                del condition.target[idx]
-
-    def add_to_tree_structure(self,
-                              variableobj,
-                              space,
-                              child,
-                              ):  # pylint: disable=R0201
-        if isinstance(variableobj, self.Redefinable):
-            name = variableobj.name
-            if child.tag == 'family':
-                name = normalize_family(name)
-            getattr(space, child.tag)[name] = variableobj
-        elif isinstance(variableobj, self.UnRedefinable):
-            getattr(space, child.tag).append(variableobj)
-        else:
-            setattr(space, child.tag, variableobj)
-
-    def set_text_to_obj(self,
                  child,
                  variableobj,
-                        ):
+                 ) -> None:
-        if child.text is None:
+        """set text
-            text = None
+        """
-        else:
+        if child.text is None or child.tag in self.forced_text_elts_as_name:
+            return
         text = child.text.strip()
         if text:
-            if child.tag in self.forced_text_elts_as_name:
-                variableobj.name = text
-            else:
             variableobj.text = text
 
-    def set_xml_attributes_to_obj(self,
+    def set_attributes(self,
                        xmlfile,
                        child,
                        variableobj,
                        ):
-        redefine = self.convert_boolean(child.attrib.get('redefine', False))
+        """ set attributes to an object
+        """
+        redefine = convert_boolean(child.attrib.get('redefine', False))
+        if redefine and child.tag == 'variable':
+            # delete old values
             has_value = hasattr(variableobj, 'value')
-        if redefine is True and child.tag == 'variable' and has_value and len(child) != 0:
+            if has_value and len(child) != 0:
                 del variableobj.value
         for attr, val in child.attrib.items():
             if redefine and attr in UNREDEFINABLE:
-                # UNREDEFINABLE concerns only 'variable' node so we can fix name
+                msg = _(f'cannot redefine attribute "{attr}" for variable "{child.attrib["name"]}"'
-                # to child.attrib['name']
+                        f' in "{xmlfile}", already defined')
-                name = child.attrib['name']
+                raise DictConsistencyError(msg, 48, variableobj.xmlfiles[:-1])
-                xmlfiles = self.display_xmlfiles(variableobj.xmlfiles[:-1])
-                raise DictConsistencyError(_(f'cannot redefine attribute "{attr}" for variable "{name}" in "{xmlfile}", already defined in {xmlfiles}'))
             if attr in self.booleans_attributs:
-                val = self.convert_boolean(val)
+                val = convert_boolean(val)
-            if not (attr == 'name' and getattr(variableobj, 'name', None) != None):
+            if attr == 'name' and getattr(variableobj, 'name', None):
+                # do not redefine name
+                continue
             setattr(variableobj, attr, val)
-        keys = list(vars(variableobj).keys())
 
-    def variableobj_tree_visitor(self,
+    def remove(self,
                child,
                variableobj,
-                                 namespace,
+               redefine_variables,
                ):
-        """Creole object tree manipulations
+        """Rougail object tree manipulations
         """
         if child.tag == 'variable':
             if child.attrib.get('remove_check', False):
@@ -518,64 +417,95 @@ class CreoleObjSpace:
             if child.attrib.get('remove_fill', False):
                 self.remove_fill(variableobj.name)
         if child.tag == 'fill':
-            # if variable is a redefine in current dictionary
+            for target in child:
-            # XXX not working with variable not in variable and in leader/followers
+                if target.tag == 'target' and target.text in redefine_variables:
-            variableobj.redefine = child.attrib['target'] in self.redefine_variables
+                    self.remove_fill(target.text)
-            if child.attrib['target'] in self.redefine_variables and child.attrib['target'] not in self.fill_removed:
-                self.remove_fill(child.attrib['target'])
-                self.fill_removed.append(child.attrib['target'])
-        if not hasattr(variableobj, 'index'):
-            variableobj.index = self.index
-        if child.tag == 'check' and child.attrib['target'] in self.redefine_variables and child.attrib['target'] not in self.check_removed:
-            self.remove_check(child.attrib['target'])
-            self.check_removed.append(child.attrib['target'])
-        if child.tag == 'condition' and child.attrib['source'] in self.redefine_variables and child.attrib['source'] not in self.condition_removed:
-            self.remove_condition(child.attrib['source'])
-            self.condition_removed.append(child.attrib['source'])
-        variableobj.namespace = namespace
 
-    def fill_variableobj_path_attribute(self,
+    def remove_check(self, name):
-                                        space,
+        """Remove a check with a specified target
-                                        child,
+        """
+        remove_checks = []
+        for idx, check in enumerate(self.space.constraints.check):  # pylint: disable=E1101
+            for target in check.target:
+                if target.name == name:
+                    remove_checks.append(idx)
+        remove_checks.sort(reverse=True)
+        for idx in remove_checks:
+            self.space.constraints.check.pop(idx)  # pylint: disable=E1101
+
+    def remove_condition(self,
+                         name: str,
+                         ) -> None:
+        """Remove a condition with a specified source
+        """
+        remove_conditions = []
+        for idx, condition in enumerate(self.space.constraints.condition):  # pylint: disable=E1101
+            if condition.source == name:
+                remove_conditions.append(idx)
+        remove_conditions.sort(reverse=True)
+        for idx in remove_conditions:
+            del self.space.constraints.condition[idx]  # pylint: disable=E1101
+
+    def remove_fill(self,
+                    name: str,
+                    ) -> None:
+        """Remove a fill with a specified target
+        """
+        remove_fills = []
+        for idx, fill in enumerate(self.space.constraints.fill):  # pylint: disable=E1101
+            for target in fill.target:
+                if target.name == name:
+                    remove_fills.append(idx)
+        remove_fills.sort(reverse=True)
+        for idx in remove_fills:
+            self.space.constraints.fill.pop(idx)  # pylint: disable=E1101
+
+    def set_path(self,
                  namespace,
                  document,
                  variableobj,
-                                        ):  # pylint: disable=R0913
+                 space,
+                 ):
         """Fill self.paths attributes
         """
-        if isinstance(space, self.help):  # pylint: disable=E1101
+        if isinstance(variableobj, self.variable):  # pylint: disable=E1101
-            return
+            if 'name' in document.attrib:
-        if child.tag == 'variable':
+                family_name = document.attrib['name']
-            family_name = normalize_family(document.attrib['name'])
-            self.paths.add_variable(namespace,
-                                    child.attrib['name'],
-                                    family_name,
-                                    document.attrib.get('dynamic') != None,
-                                    variableobj)
-            if child.attrib.get('redefine', 'False') == 'True':
-                if namespace == Config['variable_namespace']:
-                    self.redefine_variables.append(child.attrib['name'])
             else:
-                    self.redefine_variables.append(namespace + '.' + family_name + '.' +
+                family_name = namespace
-                                                   child.attrib['name'])
 
-        elif child.tag == 'family':
+            self.paths.add_variable(namespace,
-            family_name = normalize_family(child.attrib['name'])
+                                    variableobj.name,
-            if namespace != Config['variable_namespace']:
+                                    space.path,
+                                    document.attrib.get('dynamic') is not None,
+                                    variableobj,
+                                    )
+        elif isinstance(variableobj, self.family):  # pylint: disable=E1101
+            family_name = variableobj.name
+            if namespace != self.rougailconfig['variable_namespace']:
                 family_name = namespace + '.' + family_name
             self.paths.add_family(namespace,
                                   family_name,
                                   variableobj,
+                                  space.path,
                                   )
-            variableobj.path = self.paths.get_family_path(family_name, namespace)
+        elif isinstance(variableobj, self.variables):
+            variableobj.path = variableobj.name
 
-    def space_visitor(self, eosfunc_file):  # pylint: disable=C0111
+    @staticmethod
-        self.funcs_path = eosfunc_file
+    def add_to_tree_structure(variableobj,
-        SpaceAnnotator(self, eosfunc_file)
+                              space,
-
+                              child,
-    def save(self,
+                              namespace: str,
-             ):
+                              ) -> None:
-        tiramisu_objects = TiramisuReflector(self.space,
+        """add a variable to the tree
-                                             self.funcs_path,
+        """
-                                             )
+        variableobj.namespace = namespace
-        return tiramisu_objects.get_text() + '\n'
+        if isinstance(variableobj, Redefinable):
+            name = variableobj.name
+            tag = FORCE_TAG.get(child.tag, child.tag)
+            getattr(space, tag)[name] = variableobj
+        elif isinstance(variableobj, UnRedefinable):
+            getattr(space, child.tag).append(variableobj)
+        else:
+            setattr(space, child.tag, variableobj)

src/rougail/path.py

@@ -1,209 +1,250 @@
+"""Manage path to find objects
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from typing import List
 from .i18n import _
+from .error import DictConsistencyError
 from .utils import normalize_family
-from .error import OperationError, DictConsistencyError
-from .config import Config
 
 
 class Path:
-    """Helper class to handle the `path` attribute of a CreoleObjSpace
+    """Helper class to handle the `path` attribute.
-    instance.
 
     sample: path="creole.general.condition"
     """
-    def __init__(self):
+    def __init__(self,
+                 rougailconfig: 'RougailConfig',
+                 ) -> None:
         self.variables = {}
         self.families = {}
         self.full_paths_families = {}
         self.full_paths_variables = {}
+        self.variable_namespace = rougailconfig['variable_namespace']
 
     # Family
     def add_family(self,
                    namespace: str,
                    name: str,
                    variableobj: str,
+                   subpath: str,
                    ) -> str:  # pylint: disable=C0111
-        if '.' not in name and namespace == Config['variable_namespace']:
+        """Add a new family
-            full_name = '.'.join([namespace, name])
+        """
+        if namespace == self.variable_namespace:
+            full_name = '.'.join([subpath, name])
+            if name in self.full_paths_families:
+                msg = _(f'Duplicate family name "{name}"')
+                raise DictConsistencyError(msg, 55, variableobj.xmlfiles)
             self.full_paths_families[name] = full_name
         else:
+            if '.' not in name:  # pragma: no cover
+                msg = _(f'Variable "{name}" in namespace "{namespace}" must have dot')
+                raise DictConsistencyError(msg, 39, variableobj.xmlfiles)
             full_name = name
-        if full_name in self.families and self.families[full_name]['variableobj'] != variableobj:
+        if full_name in self.families and \
-            raise DictConsistencyError(_(f'Duplicate family name {name}'))
+                self.families[full_name]['variableobj'] != variableobj:  # pragma: no cover
+            msg = _(f'Duplicate family name "{name}"')
+            raise DictConsistencyError(msg, 37, variableobj.xmlfiles)
+        if full_name in self.variables:
+            msg = _(f'A variable and a family has the same path "{full_name}"')
+            raise DictConsistencyError(msg, 56, variableobj.xmlfiles)
         self.families[full_name] = dict(name=name,
                                         namespace=namespace,
                                         variableobj=variableobj,
                                         )
+        variableobj.path = full_name
 
-    def get_family_path(self,
+    def add_leadership(self,
+                       namespace: str,
+                       path: str,
+                       variableobj: str,
+                       ) -> str:  # pylint: disable=C0111
+        """add a new leadership
+        """
+        self.families[path] = dict(name=path,
+                                   namespace=namespace,
+                                   variableobj=variableobj,
+                                   )
+        variableobj.path = path
+
+    def get_family(self,
                    name: str,
                    current_namespace: str,
-                        ) -> str:  # pylint: disable=C0111
-        name = normalize_family(name,
-                                check_name=False,
-                                allow_dot=True,
-                                )
-        if '.' not in name and current_namespace == Config['variable_namespace'] and name in self.full_paths_families:
-            name = self.full_paths_families[name]
-        if current_namespace is None:  # pragma: no cover
-            raise OperationError('current_namespace must not be None')
-        dico = self.families[name]
-        if dico['namespace'] != Config['variable_namespace'] and current_namespace != dico['namespace']:
-            raise DictConsistencyError(_('A family located in the {} namespace '
-                                               'shall not be used in the {} namespace').format(
-                                                   dico['namespace'], current_namespace))
-        return dico['name']
-
-    def get_family_obj(self,
-                       name: str,
                    ) -> 'Family':  # pylint: disable=C0111
-        if '.' not in name and name in self.full_paths_families:
+        """Get a family
+        """
+        name = '.'.join([normalize_family(subname) for subname in name.split('.')])
+        if name not in self.families and name in self.full_paths_families:
             name = self.full_paths_families[name]
         if name not in self.families:
-            raise DictConsistencyError(_('unknown family {}').format(name))
+            raise DictConsistencyError(_(f'unknown option {name}'), 42, [])
         dico = self.families[name]
+        if current_namespace not in [self.variable_namespace, 'services'] and \
+                current_namespace != dico['namespace']:
+            msg = _(f'A family located in the "{dico["namespace"]}" namespace '
+                    f'shall not be used in the "{current_namespace}" namespace')
+            raise DictConsistencyError(msg, 38, [])
         return dico['variableobj']
 
-    def family_is_defined(self,
-                        name: str,
-                        ) -> str:  # pylint: disable=C0111
-        if '.' not in name and name not in self.families and name in self.full_paths_families:
-            return True
-        return name in self.families
-
     # Leadership
     def set_leader(self,
                    namespace: str,
                    leader_family_name: str,
+                   leadership_name: str,
                    name: str,
-                   leader_name: str,
                    ) -> None:  # pylint: disable=C0111
+        """set a variable a leadership member
+        """
         # need rebuild path and move object in new path
-        old_path = namespace + '.' + leader_family_name + '.' + name
+        old_path = leader_family_name + '.' + name
-        dico = self._get_variable(old_path)
+        leadership_path = leader_family_name + '.' + leadership_name
-        del self.variables[old_path]
+        new_path = leadership_path + '.' + name
-        new_path = namespace + '.' + leader_family_name + '.' + leader_name + '.' + name
+        self.variables[new_path] = self.variables.pop(old_path)
-        self.add_variable(namespace,
+        self.variables[new_path]['leader'] = leadership_path
-                          new_path,
+        self.variables[new_path]['variableobj'].path = new_path
-                          dico['family'],
+        self.variables[new_path]['family'] = leadership_path
-                          False,
+        if namespace == self.variable_namespace:
-                          dico['variableobj'],
-                          )
-        if namespace == Config['variable_namespace']:
             self.full_paths_variables[name] = new_path
-        else:
-            name = new_path
-        dico = self._get_variable(name)
-        if dico['leader'] != None:
-            raise DictConsistencyError(_('Already defined leader {} for variable'
-                                         ' {}'.format(dico['leader'], name)))
-        dico['leader'] = leader_name
 
-    def get_leader(self, name):  # pylint: disable=C0111
+    def is_leader(self, path):  # pylint: disable=C0111
-        return self._get_variable(name)['leader']
+        """Is the variable is a leader
+        """
+        variable = self._get_variable(path)
+        if not variable['leader']:
+            return False
+        leadership = self.get_family(variable['leader'], variable['variableobj'].namespace)
+        return leadership.variable[0].path == path
+
+    def is_follower(self, path):
+        """Is the variable is a follower
+        """
+        variable = self._get_variable(path)
+        if not variable['leader']:
+            return False
+        leadership = self.get_family(variable['leader'], variable['variableobj'].namespace)
+        return leadership.variable[0].path != path
 
     # Variable
-    def add_variable(self,
+    def add_variable(self,  # pylint: disable=R0913
                      namespace: str,
                      name: str,
                      family: str,
                      is_dynamic: bool,
                      variableobj,
                      ) -> str:  # pylint: disable=C0111
+        """Add a new variable (with path)
+        """
         if '.' not in name:
-            full_name = '.'.join([namespace, family, name])
+            full_path = '.'.join([family, name])
-            self.full_paths_variables[name] = full_name
+            if namespace == self.variable_namespace:
+                self.full_paths_variables[name] = full_path
         else:
-            full_name = name
+            full_path = name
-            if namespace == Config['variable_namespace']:
+        variableobj.path = full_path
-                name = name.rsplit('.', 1)[1]
+        if full_path in self.families:
-        self.variables[full_name] = dict(name=name,
+            msg = _(f'A family and a variable has the same path "{full_path}"')
+            raise DictConsistencyError(msg, 57, variableobj.xmlfiles)
+        self.variables[full_path] = dict(name=name,
                                          family=family,
-                                         namespace=namespace,
                                          leader=None,
                                          is_dynamic=is_dynamic,
-                                         variableobj=variableobj)
+                                         variableobj=variableobj,
+                                         )
 
-    def get_variable_name(self,
+    def get_variable(self,
-                          name,
+                     name: str,
-                          ):  # pylint: disable=C0111
+                     xmlfiles: List[str]=[],
-        return self._get_variable(name)['name']
-
-    def get_variable_obj(self,
-                         name:str,
                      ) -> 'Variable':  # pylint: disable=C0111
-        return self._get_variable(name)['variableobj']
+        """Get variable object from a path
+        """
+        variable, suffix = self._get_variable(name, with_suffix=True, xmlfiles=xmlfiles)
+        if suffix:
+            raise DictConsistencyError(_(f"{name} is a dynamic variable"), 36, [])
+        return variable['variableobj']
 
-    def get_variable_family_name(self,
+    def get_variable_family_path(self,
                                  name: str,
+                                 xmlfiles: List[str]=False,
                                  ) -> str:  # pylint: disable=C0111
-        return self._get_variable(name)['family']
+        """Get the full path of a family
-
+        """
-    def get_variable_namespace(self,
+        return self._get_variable(name, xmlfiles=xmlfiles)['family']
-                               name: str,
-                               ) -> str:  # pylint: disable=C0111
-        return self._get_variable(name)['namespace']
 
     def get_variable_path(self,
                           name: str,
                           current_namespace: str,
-                          allow_source: str=False,
+                          xmlfiles: List[str]=[],
-                          with_suffix: bool=False,
                           ) -> str:  # pylint: disable=C0111
-        if current_namespace is None:  # pragma: no cover
+        """get full path of a variable
-            raise OperationError('current_namespace must not be None')
+        """
-        if with_suffix:
         dico, suffix = self._get_variable(name,
                                           with_suffix=True,
+                                          xmlfiles=xmlfiles,
                                           )
-        else:
+        namespace = dico['variableobj'].namespace
-            dico = self._get_variable(name)
+        if namespace not in [self.variable_namespace, 'services'] and \
-        if not allow_source:
+                current_namespace != namespace:
-            if dico['namespace'] not in [Config['variable_namespace'], 'services'] and current_namespace != dico['namespace']:
+            msg = _(f'A variable located in the "{namespace}" namespace shall not be used '
-                raise DictConsistencyError(_('A variable located in the {} namespace '
+                    f'in the "{current_namespace}" namespace')
-                                                   'shall not be used in the {} namespace').format(
+            raise DictConsistencyError(msg, 41, xmlfiles)
-                                                       dico['namespace'], current_namespace))
+        return dico['variableobj'].path, suffix
-        if '.' in dico['name']:
-            value = dico['name']
-        else:
-            list_path = [dico['namespace'], dico['family']]
-            if dico['leader'] is not None:
-                list_path.append(dico['leader'])
-            list_path.append(dico['name'])
-            value = '.'.join(list_path)
-        if with_suffix:
-            return value, suffix
-        return value
 
     def path_is_defined(self,
-                        name: str,
+                        path: str,
                         ) -> str:  # pylint: disable=C0111
-        if '.' not in name and name not in self.variables and name in self.full_paths_variables:
+        """The path is a valid path
+        """
+        if '.' not in path and path not in self.variables and path in self.full_paths_variables:
             return True
-        return name in self.variables
+        return path in self.variables
+
+    def variable_is_dynamic(self,
+                            name: str,
+                            ) -> bool:
+        """This variable is in dynamic family
+        """
+        return self._get_variable(name)['is_dynamic']
 
     def _get_variable(self,
                       name: str,
                       with_suffix: bool=False,
+                      xmlfiles: List[str]=[],
                       ) -> str:
-        if name not in self.variables:
+        name = '.'.join([normalize_family(subname) for subname in name.split('.')])
         if name not in self.variables:
             if '.' not in name and name in self.full_paths_variables:
                 name = self.full_paths_variables[name]
-            if name not in self.variables:
+            elif with_suffix:
-                for var_name, variable in self.variables.items():
+                for var_name, full_path in self.full_paths_variables.items():
-                    if variable['is_dynamic'] and name.startswith(var_name):
-                        if not with_suffix:
-                            raise Exception('This option is dynamic, should use "with_suffix" attribute')
-                        return variable, name[len(var_name):]
-                if '.' not in name:
-                    for var_name, path in self.full_paths_variables.items():
                     if name.startswith(var_name):
-                            variable = self.variables[self.full_paths_variables[var_name]]
+                        variable = self._get_variable(full_path)
                         if variable['is_dynamic']:
-                                if not with_suffix:
-                                    raise Exception('This option is dynamic, should use "with_suffix" attribute')
                             return variable, name[len(var_name):]
-                raise DictConsistencyError(_('unknown option {}').format(name))
+        if name not in self.variables:
+            raise DictConsistencyError(_(f'unknown option "{name}"'), 42, xmlfiles)
         if with_suffix:
             return self.variables[name], None
         return self.variables[name]

src/rougail/template.py

@@ -1,452 +0,0 @@
-# -*- coding: utf-8 -*-
-"""
-Gestion du mini-langage de template
-On travaille sur les fichiers cibles
-"""
-
-import imp
-from shutil import copy
-import logging
-from typing import Dict, Any
-from subprocess import call
-from os import listdir, makedirs, getcwd, chdir
-from os.path import dirname, join, isfile, abspath, normpath, relpath
-
-from Cheetah.Template import Template as ChtTemplate
-from Cheetah.NameMapper import NotFound as CheetahNotFound
-
-try:
-    from tiramisu3 import Config
-    from tiramisu3.error import PropertiesOptionError
-except:
-    from tiramisu import Config
-    from tiramisu.error import PropertiesOptionError
-
-from .config import Config
-from .error import FileNotFound, TemplateError
-from .i18n import _
-from .utils import normalize_family
-
-
-log = logging.getLogger(__name__)
-log.addHandler(logging.NullHandler())
-
-
-class IsDefined:
-    """
-    filtre permettant de ne pas lever d'exception au cas où
-    la variable Creole n'est pas définie
-     """
-    def __init__(self, context):
-        self.context = context
-
-    def __call__(self, varname):
-        if '.' in varname:
-            splitted_var = varname.split('.')
-            if len(splitted_var) != 2:
-                msg = _("Group variables must be of type leader.follower")
-                raise KeyError(msg)
-            leader, follower = splitted_var
-            if leader in self.context:
-                return follower in self.context[leader].follower.keys()
-            return False
-        else:
-            return varname in self.context
-
-
-@classmethod
-def cl_compile(kls, *args, **kwargs):
-    kwargs['compilerSettings'] = {'directiveStartToken' : '%',
-                                  'cheetahVarStartToken' : '%%',
-                                  'EOLSlurpToken' : '%',
-                                  'PSPStartToken' : 'µ' * 10,
-                                  'PSPEndToken' : 'µ' * 10,
-                                  'commentStartToken' : 'µ' * 10,
-                                  'commentEndToken' : 'µ' * 10,
-                                  'multiLineCommentStartToken' : 'µ' * 10,
-                                  'multiLineCommentEndToken' : 'µ' * 10}
-    return kls.old_compile(*args, **kwargs)
-ChtTemplate.old_compile = ChtTemplate.compile
-ChtTemplate.compile = cl_compile
-
-
-class CheetahTemplate(ChtTemplate):
-    """classe pour personnaliser et faciliter la construction
-    du template Cheetah
-    """
-    def __init__(self,
-                 filename: str,
-                 context,
-                 eosfunc: Dict,
-                 destfilename,
-                 variable,
-                 ):
-        """Initialize Creole CheetahTemplate
-        """
-        extra_context = {'is_defined' : IsDefined(context),
-                         'normalize_family': normalize_family,
-                         'rougail_filename': destfilename
-                         }
-        if variable:
-            extra_context['rougail_variable'] = variable
-        ChtTemplate.__init__(self,
-                             file=filename,
-                             searchList=[context, eosfunc, extra_context])
-
-    # FORK of Cheetah fonction, do not replace '\\' by '/'
-    def serverSidePath(self,
-                       path=None,
-                       normpath=normpath,
-                       abspath=abspath
-                       ):
-
-        # strange...
-        if path is None and isinstance(self, str):
-            path = self
-        if path:
-            return normpath(abspath(path))
-#            return normpath(abspath(path.replace("\\", '/')))
-        elif hasattr(self, '_filePath') and self._filePath:
-            return normpath(abspath(self._filePath))
-        else:
-            return None
-
-
-class CreoleLeader:
-    def __init__(self, value, follower=None, index=None):
-        """
-        On rend la variable itérable pour pouvoir faire:
-        for ip in iplist:
-        print(ip.network)
-        print(ip.netmask)
-        print(ip)
-        index is used for CreoleLint
-        """
-        self._value = value
-        if follower is not None:
-            self.follower = follower
-        else:
-            self.follower = {}
-        self._index = index
-
-    def __getattr__(self, name):
-        """Get follower variable or attribute of leader value.
-
-        If the attribute is a name of a follower variable, return its value.
-        Otherwise, returns the requested attribute of leader value.
-        """
-        if name in self.follower:
-            value = self.follower[name]
-            if isinstance(value, PropertiesOptionError):
-                raise AttributeError()
-            return value
-        else:
-            return getattr(self._value, name)
-
-    def __getitem__(self, index):
-        """Get a leader.follower at requested index.
-        """
-        ret = {}
-        for key, values in self.follower.items():
-            ret[key] = values[index]
-        return CreoleLeader(self._value[index], ret, index)
-
-    def __iter__(self):
-        """Iterate over leader.follower.
-
-        Return synchronised value of leader.follower.
-        """
-        for i in range(len(self._value)):
-            ret = {}
-            for key, values in self.follower.items():
-                ret[key] = values[i]
-            yield CreoleLeader(self._value[i], ret, i)
-
-    def __len__(self):
-        """Delegate to leader value
-        """
-        return len(self._value)
-
-    def __repr__(self):
-        """Show CreoleLeader as dictionary.
-
-        The leader value is stored under 'value' key.
-        The followers are stored under 'follower' key.
-        """
-        return repr({'value': self._value, 'follower': self.follower})
-
-    def __eq__(self, value):
-        return value == self._value
-
-    def __ne__(self, value):
-        return value != self._value
-
-    def __lt__(self, value):
-        return self._value < value
-
-    def __le__(self, value):
-        return self._value <= value
-
-    def __gt__(self, value):
-        return self._value > value
-
-    def __ge__(self, value):
-        return self._value >= value
-
-    def __str__(self):
-        """Delegate to leader value
-        """
-        return str(self._value)
-
-    def __add__(self, val):
-        return self._value.__add__(val)
-
-    def __radd__(self, val):
-        return val + self._value
-
-    def __contains__(self, item):
-        return item in self._value
-
-    async def add_follower(self, config, name, path):
-        if isinstance(self._value, list):
-            values = []
-            for idx in range(len(self._value)):
-                try:
-                    values.append(await config.option(path, idx).value.get())
-                except PropertiesOptionError as err:
-                    values.append(err)
-        else:
-            raise Exception('hu?')
-        self.follower[name] = values
-
-
-class CreoleExtra:
-    def __init__(self,
-                 suboption: Dict) -> None:
-        self.suboption = suboption
-
-    def __getattr__(self,
-                    key: str) -> Any:
-        return self.suboption[key]
-
-    def __repr__(self):
-        return self.suboption.__str__()
-
-    def __iter__(self):
-        return iter(self.suboption.values())
-
-
-class CreoleTemplateEngine:
-    """Engine to process Creole cheetah template
-    """
-    def __init__(self,
-                 config: Config,
-                 eosfunc_file: str,
-                 distrib_dir: str,
-                 tmp_dir: str,
-                 dest_dir: str,
-                 ) -> None:
-        self.config = config
-        self.dest_dir = dest_dir
-        self.tmp_dir = tmp_dir
-        self.distrib_dir = distrib_dir
-        eos = {}
-        if eosfunc_file is not None:
-            eosfunc = imp.load_source('eosfunc', eosfunc_file)
-            for func in dir(eosfunc):
-                 if not func.startswith('_'):
-                     eos[func] = getattr(eosfunc, func)
-        self.eosfunc = eos
-        self.rougail_variables_dict = {}
-
-    async def load_eole_variables_rougail(self,
-                                          optiondescription):
-        for option in await optiondescription.list('all'):
-            if await option.option.isoptiondescription():
-                if await option.option.isleadership():
-                    for idx, suboption in enumerate(await option.list('all')):
-                        if idx == 0:
-                            leader = CreoleLeader(await suboption.value.get())
-                            self.rougail_variables_dict[await suboption.option.name()] = leader
-                        else:
-                            await leader.add_follower(self.config,
-                                                   await suboption.option.name(),
-                                                   await suboption.option.path())
-                else:
-                    await self.load_eole_variables_rougail(option)
-            else:
-                self.rougail_variables_dict[await option.option.name()] = await option.value.get()
-
-    async def load_eole_variables(self,
-                                  namespace,
-                                  optiondescription):
-        families = {}
-        for family in await optiondescription.list('all'):
-            variables = {}
-            for variable in await family.list('all'):
-                if await variable.option.isoptiondescription():
-                    if await variable.option.isleadership():
-                        for idx, suboption in enumerate(await variable.list('all')):
-                            if idx == 0:
-                                leader = CreoleLeader(await suboption.value.get())
-                                leader_name = await suboption.option.name()
-                            else:
-                                await leader.add_follower(self.config,
-                                                       await suboption.option.name(),
-                                                       await suboption.option.path())
-                        variables[leader_name] = leader
-                    else:
-                        subfamilies = await self.load_eole_variables(await variable.option.name(),
-                                                                     variable,
-                                                                     )
-                        variables[await variable.option.name()] = subfamilies
-                else:
-                    variables[await variable.option.name()] = await variable.value.get()
-            families[await family.option.name()] = CreoleExtra(variables)
-        return CreoleExtra(families)
-
-    def patch_template(self,
-                       filename: str,
-                       tmp_dir: str,
-                       patch_dir: str,
-                       ) -> None:
-        """Apply patch to a template
-        """
-        patch_cmd = ['patch', '-d', tmp_dir, '-N', '-p1']
-        patch_no_debug = ['-s', '-r', '-', '--backup-if-mismatch']
-
-        patch_file = join(patch_dir, f'{filename}.patch')
-        if isfile(patch_file):
-            log.info(_("Patching template '{filename}' with '{patch_file}'"))
-            rel_patch_file = relpath(patch_file, tmp_dir)
-            ret = call(patch_cmd + patch_no_debug + ['-i', rel_patch_file])
-            if ret:
-                patch_cmd_err = ' '.join(patch_cmd + ['-i', rel_patch_file])
-                log.error(_(f"Error applying patch: '{rel_patch_file}'\nTo reproduce and fix this error {patch_cmd_err}"))
-                copy(join(self.distrib_dir, filename), tmp_dir)
-
-    def prepare_template(self,
-                         filename: str,
-                         tmp_dir: str,
-                         patch_dir: str,
-                         ) -> None:
-        """Prepare template source file
-        """
-        log.info(_("Copy template: '{filename}' -> '{tmp_dir}'"))
-        copy(filename, tmp_dir)
-        self.patch_template(filename, tmp_dir, patch_dir)
-
-    def process(self,
-                source: str,
-                true_destfilename: str,
-                destfilename: str,
-                filevar: Dict,
-                variable: Any,
-                ):
-        """Process a cheetah template
-        """
-        # full path of the destination file
-        log.info(_(f"Cheetah processing: '{destfilename}'"))
-        try:
-            cheetah_template = CheetahTemplate(source,
-                                               self.rougail_variables_dict,
-                                               self.eosfunc,
-                                               true_destfilename,
-                                               variable,
-                                               )
-            data = str(cheetah_template)
-        except CheetahNotFound as err:
-            varname = err.args[0][13:-1]
-            raise TemplateError(_(f"Error: unknown variable used in template {source} to {destfilename} : {varname}"))
-        except Exception as err:
-            raise TemplateError(_(f"Error while instantiating template {source} to {destfilename}: {err}"))
-
-        with open(destfilename, 'w') as file_h:
-            file_h.write(data)
-
-    def instance_file(self,
-                      filevar: Dict,
-                      service_name: str,
-                      tmp_dir: str,
-                      dest_dir: str,
-                      ) -> None:
-        """Run templatisation on one file
-        """
-        log.info(_("Instantiating file '{filename}'"))
-        if 'variable' in filevar:
-            variable = filevar['variable']
-        else:
-            variable = None
-        filenames = filevar['name']
-        if not isinstance(filenames, list):
-            filenames = [filenames]
-            if variable:
-                variable = [variable]
-        for idx, filename in enumerate(filenames):
-            destfilename = join(dest_dir, filename[1:])
-            makedirs(dirname(destfilename), exist_ok=True)
-            if variable:
-                var = variable[idx]
-            else:
-                var = None
-            source = join(tmp_dir, filevar['source'])
-            if filevar['templating']:
-                self.process(source,
-                             filename,
-                             destfilename,
-                             filevar,
-                             var)
-            else:
-                copy(source, destfilename)
-
-    async def instance_files(self) -> None:
-        """Run templatisation on all files
-        """
-        ori_dir = getcwd()
-        tmp_dir = relpath(self.tmp_dir, self.distrib_dir)
-        dest_dir = relpath(self.dest_dir, self.distrib_dir)
-        patch_dir = relpath(Config['patch_dir'], self.distrib_dir)
-        chdir(self.distrib_dir)
-        for option in await self.config.option.list(type='all'):
-            namespace = await option.option.name()
-            if namespace == Config['variable_namespace']:
-                await self.load_eole_variables_rougail(option)
-            else:
-                families = await self.load_eole_variables(namespace,
-                                                          option)
-                self.rougail_variables_dict[namespace] = families
-        for template in listdir('.'):
-            self.prepare_template(template, tmp_dir, patch_dir)
-        for service_obj in await self.config.option('services').list('all'):
-            service_name = await service_obj.option.doc()
-            for fills in await service_obj.list('all'):
-                if await fills.option.name() in ['files', 'overrides']:
-                    for fill_obj in await fills.list('all'):
-                        fill = await fill_obj.value.dict()
-                        filename = fill['source']
-                        if not isfile(filename):
-                            raise FileNotFound(_(f"File {filename} does not exist."))
-                        if fill.get('activate', False):
-                            self.instance_file(fill,
-                                               service_name,
-                                               tmp_dir,
-                                               dest_dir,
-                                               )
-                        else:
-                            log.debug(_("Instantiation of file '{filename}' disabled"))
-        chdir(ori_dir)
-
-
-async def generate(config: Config,
-                   eosfunc_file: str,
-                   distrib_dir: str,
-                   tmp_dir: str,
-                   dest_dir: str,
-                   ) -> None:
-    engine = CreoleTemplateEngine(config,
-                                  eosfunc_file,
-                                  distrib_dir,
-                                  tmp_dir,
-                                  dest_dir,
-                                  )
-    await engine.instance_files()

src/rougail/template/base.py

@@ -0,0 +1,385 @@
+"""Template langage for Rougail
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+
+from shutil import copy
+import logging
+from typing import Dict, Any
+from subprocess import call
+from os import listdir, makedirs, getcwd, chdir
+from os.path import dirname, join, isfile, isdir, abspath
+
+
+try:
+    from tiramisu3 import Config
+    from tiramisu3.error import PropertiesOptionError  # pragma: no cover
+except ModuleNotFoundError:  # pragma: no cover
+    from tiramisu import Config
+    from tiramisu.error import PropertiesOptionError
+
+from ..config import RougailConfig
+from ..error import FileNotFound, TemplateError
+from ..i18n import _
+from ..utils import load_modules
+
+from . import engine as engines
+ENGINES = {}
+for engine in engines.__all__:
+    ENGINES[engine] = getattr(engines, engine)
+
+
+log = logging.getLogger(__name__)
+log.addHandler(logging.NullHandler())
+
+
+class RougailLeaderIndex:
+    """This object is create when access to a specified Index of the variable
+    """
+    def __init__(self,
+                 value,
+                 follower,
+                 index,
+                 ) -> None:
+        self._value = value
+        self._follower = follower
+        self._index = index
+
+    def __getattr__(self, name):
+        if name not in self._follower:
+            raise AttributeError()
+        value = self._follower[name]
+        if isinstance(value, PropertiesOptionError):
+            raise AttributeError()
+        return value
+
+    def __str__(self):
+        return str(self._value)
+
+    def __lt__(self, value):
+        return self._value.__lt__(value)
+
+    def __le__(self, value):
+        return self._value.__le__(value)
+
+    def __eq__(self, value):
+        return self._value.__eq__(value)
+
+    def __ne__(self, value):
+        return self._value.__ne__(value)
+
+    def __gt__(self, value):
+        return self._value.__gt__(value)
+
+    def __ge__(self, value):
+        return self._value >= value
+
+    def __add__(self, value):
+        return self._value.__add__(value)
+
+    def __radd__(self, value):
+        return value + self._value
+
+
+class RougailLeader:
+    """Implement access to leader and follower variable
+    For examples: %%leader, %%leader[0].follower1
+    """
+    def __init__(self,
+                 value,
+                 ) -> None:
+        self._value = value
+        self._follower = {}
+
+    def __getitem__(self, index):
+        """Get a leader.follower at requested index.
+        """
+        followers = {key: values[index] for key, values in self._follower.items()}
+        return RougailLeaderIndex(self._value[index],
+                                 followers,
+                                 index,
+                                 )
+
+    def __iter__(self):
+        """Iterate over leader.follower.
+
+        Return synchronised value of leader.follower.
+        """
+        for index in range(len(self._value)):
+            yield self.__getitem__(index)
+
+    def __len__(self):
+        return len(self._value)
+
+    def __contains__(self, value):
+        return self._value.__contains__(value)
+
+    async def add_follower(self,
+                           config,
+                           name: str,
+                           path: str,
+                           ):
+        """Add a new follower
+        """
+        self._follower[name] = []
+        for index in range(len(self._value)):
+            try:
+                value = await config.option(path, index).value.get()
+            except PropertiesOptionError as err:
+                value = err
+            self._follower[name].append(value)
+
+
+class RougailExtra:
+    """Object that implement access to extra variable
+    For example %%extra1.family.variable
+    """
+    def __init__(self,
+                 suboption: Dict) -> None:
+        self.suboption = suboption
+
+    def __getattr__(self,
+                    key: str,
+                    ) -> Any:
+        try:
+            return self.suboption[key]
+        except KeyError:
+            raise AttributeError(f'unable to find extra "{key}"')
+
+    def __iter__(self):
+        return iter(self.suboption.values())
+
+    def items(self):
+        return self.suboption.items()
+
+
+class RougailBaseTemplate:
+    """Engine to process Creole cheetah template
+    """
+    def __init__(self,  # pylint: disable=R0913
+                 config: Config,
+                 rougailconfig: RougailConfig=None,
+                 ) -> None:
+        if rougailconfig is None:
+            rougailconfig = RougailConfig
+        self.config = config
+        self.destinations_dir = abspath(rougailconfig['destinations_dir'])
+        self.tmp_dir = abspath(rougailconfig['tmp_dir'])
+        self.templates_dir = abspath(rougailconfig['templates_dir'])
+        self.patches_dir = abspath(rougailconfig['patches_dir'])
+        eos = {}
+        functions_file = rougailconfig['functions_file']
+        if isfile(functions_file):
+            eosfunc = load_modules(functions_file)
+            for func in dir(eosfunc):
+                if not func.startswith('_'):
+                    eos[func] = getattr(eosfunc, func)
+        self.eosfunc = eos
+        self.rougail_variables_dict = {}
+        self.rougailconfig = rougailconfig
+        self.log = log
+        self.engines = ENGINES
+
+    def patch_template(self,
+                       filename: str,
+                       ) -> None:
+        """Apply patch to a template
+        """
+        patch_cmd = ['patch', '-d', self.tmp_dir, '-N', '-p1', '-f']
+        patch_no_debug = ['-s', '-r', '-', '--backup-if-mismatch']
+
+        patch_file = join(self.patches_dir, f'{filename}.patch')
+        if isfile(patch_file):
+            self.log.info(_("Patching template '{filename}' with '{patch_file}'"))
+            ret = call(patch_cmd + patch_no_debug + ['-i', patch_file])
+            if ret:  # pragma: no cover
+                patch_cmd_err = ' '.join(patch_cmd + ['-i', patch_file])
+                msg = _(f"Error applying patch: '{patch_file}'\n"
+                        f"To reproduce and fix this error {patch_cmd_err}")
+                self.log.error(_(msg))
+                copy(join(self.templates_dir, filename), self.tmp_dir)
+
+    def prepare_template(self,
+                         filename: str,
+                         ) -> None:
+        """Prepare template source file
+        """
+        self.log.info(_("Copy template: '{filename}' -> '{self.tmp_dir}'"))
+        if not isdir(self.tmp_dir):
+            raise TemplateError(_(f'cannot find tmp_dir {self.tmp_dir}'))
+        copy(filename, self.tmp_dir)
+        self.patch_template(filename)
+
+    def instance_file(self,
+                      filevar: Dict,
+                      type_: str,
+                      service_name: str,
+                      ) -> None:
+        """Run templatisation on one file
+        """
+        self.log.info(_("Instantiating file '{filename}'"))
+        if 'variable' in filevar:
+            variable = filevar['variable']
+        else:
+            variable = None
+        filenames = filevar.get('name')
+        if not isinstance(filenames, list):
+            filenames = [filenames]
+            if variable and not isinstance(variable, list):
+                variable = [variable]
+        if not isdir(self.destinations_dir):
+            raise TemplateError(_(f'cannot find destinations_dir {self.destinations_dir}'))
+        for idx, filename, in enumerate(filenames):
+            if variable:
+                var = variable[idx]
+            else:
+                var = None
+            func = f'_instance_{type_}'
+            data = getattr(self, func)(filevar,
+                                       filename,
+                                       service_name,
+                                       variable,
+                                       idx,
+                                       )
+            if data is None:
+                continue
+            filename, source, destfile, var = data
+            destfilename = join(self.destinations_dir, destfile[1:])
+            makedirs(dirname(destfilename), exist_ok=True)
+            self.log.info(_(f"{filevar['engine']} processing: '{destfilename}'"))
+            self.engines[filevar['engine']].process(filename=filename,
+                                                    source=source,
+                                                    true_destfilename=destfile,
+                                                    destfilename=destfilename,
+                                                    destdir=self.destinations_dir,
+                                                    variable=var,
+                                                    rougail_variables_dict=self.rougail_variables_dict,
+                                                    eosfunc=self.eosfunc,
+                                                    )
+
+    async def instance_files(self) -> None:
+        """Run templatisation on all files
+        """
+        ori_dir = getcwd()
+        chdir(self.templates_dir)
+        for option in await self.config.option.list(type='all'):
+            namespace = await option.option.name()
+            is_var_namespace = namespace == self.rougailconfig['variable_namespace']
+            self.rougail_variables_dict[namespace] = await self.load_variables(option,
+                                                                               is_var_namespace,
+                                                                               )
+        for template in listdir('.'):
+            self.prepare_template(template)
+        for included in (True, False):
+            for service_obj in await self.config.option('services').list('all'):
+                service_name = await service_obj.option.name()
+                if await service_obj.option('activate').value.get() is False:
+                    if included is False:
+                        self.desactive_service(service_name)
+                    continue
+                for fills in await service_obj.list('optiondescription'):
+                    type_ = await fills.option.name()
+                    for fill_obj in await fills.list('all'):
+                        fill = await fill_obj.value.dict()
+                        if 'included' in fill:
+                            if (fill['included'] == 'no' and included is True) or \
+                                    (fill['included'] != 'no' and included is False):
+                                continue
+                        elif included is True:
+                            continue
+                        if fill['activate']:
+                            self.instance_file(fill, type_, service_name)
+                        else:
+                            self.log.debug(_("Instantiation of file '{filename}' disabled"))
+                self.post_instance_service(service_name)
+        self.post_instance()
+        chdir(ori_dir)
+
+    def desactive_service(self,
+                          service_name: str,
+                          ):
+        raise NotImplementedError(_('cannot desactivate a service'))
+
+    def post_instance_service(self, service_name):  # pragma: no cover
+        pass
+
+    def post_instance(self):  # pragma: no cover
+        pass
+
+    def _instance_ip(self,
+                     *args,
+                     ) -> None:  # pragma: no cover
+        raise NotImplementedError(_('cannot instanciate this service type ip'))
+
+    def _instance_files(self,
+                        *args,
+                        ) -> None:  # pragma: no cover
+        raise NotImplementedError(_('cannot instanciate this service type file'))
+
+    def _instance_overrides(self,
+                            *args,
+                            ) -> None:  # pragma: no cover
+        raise NotImplementedError(_('cannot instanciate this service type override'))
+
+    async def load_variables(self,
+                             optiondescription,
+                             is_variable_namespace,
+                             ) -> RougailExtra:
+        """Load all variables and set it in RougailExtra objects
+        """
+        variables = {}
+        for option in await optiondescription.list('all'):
+            if await option.option.isoptiondescription():
+                if await option.option.isleadership():
+                    for idx, suboption in enumerate(await option.list('all')):
+                        if idx == 0:
+                            leader = RougailLeader(await suboption.value.get())
+                            leader_name = await suboption.option.name()
+                            if is_variable_namespace:
+                                self.rougail_variables_dict[await suboption.option.name()] = leader
+                        else:
+                            await leader.add_follower(self.config,
+                                                      await suboption.option.name(),
+                                                      await suboption.option.path(),
+                                                      )
+                    variables[leader_name] = leader
+                else:
+                    subfamilies = await self.load_variables(option,
+                                                            is_variable_namespace,
+                                                            )
+                    variables[await option.option.name()] = subfamilies
+            else:
+                if is_variable_namespace:
+                    value = await option.value.get()
+                    self.rougail_variables_dict[await option.option.name()] = value
+                if await option.option.issymlinkoption() and await option.option.isfollower():
+                    value = []
+                    path = await option.option.path()
+                    for index in range(await option.value.len()):
+                        value.append(await self.config.option(path, index).value.get())
+                else:
+                    value = await option.value.get()
+                variables[await option.option.name()] = value
+        return RougailExtra(variables)

src/rougail/template/engine/__init__.py

@@ -0,0 +1,4 @@
+from . import none, creole, jinja2, creole_legacy
+
+
+__all__ = ('none', 'creole', 'jinja2', 'creole_legacy')

src/rougail/template/engine/creole.py

@@ -0,0 +1,135 @@
+"""Creole engine
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from os.path import abspath, normpath
+from Cheetah.Template import Template
+from Cheetah.NameMapper import NotFound
+from typing import Dict, Any
+
+from ...i18n import _
+from ...utils import normalize_family
+from ...error import TemplateError
+
+
+@classmethod
+def cl_compile(kls, *args, **kwargs):
+    """Rewrite compile methode to force some settings
+    """
+    kwargs['compilerSettings'] = {'directiveStartToken' : '%',
+                                  'cheetahVarStartToken' : '%%',
+                                  'EOLSlurpToken' : '%',
+                                  'commentStartToken' : '#',
+                                  'multiLineCommentStartToken' : '#*',
+                                  'multiLineCommentEndToken' : '*#',
+                                  }
+    return kls.old_compile(*args, **kwargs)  # pylint: disable=E1101
+Template.old_compile = Template.compile
+Template.compile = cl_compile
+
+
+class CheetahTemplate(Template):  # pylint: disable=W0223
+    """Construct a cheetah templating object
+    """
+    def __init__(self,
+                 filename: str,
+                 source: str,
+                 context,
+                 eosfunc: Dict,
+                 extra_context: Dict,
+                 ):
+        """Initialize Creole CheetahTemplate
+        """
+        if filename is not None:
+            super().__init__(file=filename,
+                             searchList=[context, eosfunc, extra_context],
+                             )
+        else:
+            super().__init__(source=source,
+                             searchList=[context, eosfunc, extra_context],
+                             )
+
+    # FORK of Cheetah function, do not replace '\\' by '/'
+    def serverSidePath(self,
+                       path=None,
+                       normpath=normpath,
+                       abspath=abspath
+                       ):  # pylint: disable=W0621
+
+        # strange...
+        if path is None and isinstance(self, str):
+            path = self
+        if path:  # pylint: disable=R1705
+            return normpath(abspath(path))
+#           original code return normpath(abspath(path.replace("\\", '/')))
+        elif hasattr(self, '_filePath') and self._filePath:  # pragma: no cover
+            return normpath(abspath(self._filePath))
+        else:  # pragma: no cover
+            return None
+
+
+# Sync to creole_legacy.py
+def process(filename: str,
+            source: str,
+            true_destfilename: str,
+            destfilename: str,
+            destdir: str,
+            variable: Any,
+            rougail_variables_dict: Dict,
+            eosfunc: Dict,
+            ):
+    """Process a cheetah template
+    """
+    # full path of the destination file
+    try:
+        extra_context = {'normalize_family': normalize_family,
+                         'rougail_filename': true_destfilename,
+                         'rougail_destination_dir': destdir,
+                         }
+        if variable is not None:
+            extra_context['rougail_variable'] = variable
+        cheetah_template = CheetahTemplate(filename,
+                                           source,
+                                           rougail_variables_dict,
+                                           eosfunc,
+                                           extra_context,
+                                           )
+        data = str(cheetah_template)
+    except NotFound as err:  # pragma: no cover
+        varname = err.args[0][13:-1]
+        if filename:
+            msg = f"Error: unknown variable used in template {filename} to {destfilename}: {varname}"
+        else:
+            msg = f"Error: unknown variable used in file {destfilename}: {varname}"
+        raise TemplateError(_(msg)) from err
+    except Exception as err:  # pragma: no cover
+        if filename:
+            msg = _(f"Error while instantiating template {filename} to {destfilename}: {err}")
+        else:
+            msg = _(f"Error while instantiating filename {destfilename}: {err}")
+        raise TemplateError(msg) from err
+
+    with open(destfilename, 'w') as file_h:
+        file_h.write(data)

src/rougail/template/engine/creole_legacy.py

@@ -0,0 +1,157 @@
+"""Legacy Creole engine
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+
+
+from typing import Dict, Any
+from Cheetah.NameMapper import NotFound
+
+from .creole import CheetahTemplate as oriCheetahTemplate
+from ...i18n import _
+from ...utils import normalize_family
+from ...error import TemplateError
+
+
+@classmethod
+def cl_compile(kls, *args, **kwargs):
+    """Rewrite compile methode to force some settings
+    """
+    kwargs['compilerSettings'] = {'directiveStartToken' : u'%',
+                   'cheetahVarStartToken' : u'%%',
+                   'EOLSlurpToken' : u'%',
+                   'PSPStartToken' : u'µ' * 10,
+                   'PSPEndToken' : u'µ' * 10,
+                   'commentStartToken' : u'µ' * 10,
+                   'commentEndToken' : u'µ' * 10,
+                   'multiLineCommentStartToken' : u'µ' * 10,
+                   'multiLineCommentEndToken' : u'µ' * 10}
+    return kls.old_compile(*args, **kwargs)  # pylint: disable=E1101
+oriCheetahTemplate.compile = cl_compile
+
+
+class IsDefined:
+    """
+    filtre permettant de ne pas lever d'exception au cas où
+    la variable Creole n'est pas définie
+     """
+    def __init__(self, context):
+        self.context = context
+
+    def __call__(self, varname):
+        if '.' in varname:
+            splitted_var = varname.split('.')
+            if len(splitted_var) != 2:
+                msg = u"Group variables must be of type master.slave"
+                raise KeyError(msg)
+            master, slave = splitted_var
+            if master in self.context:
+                return slave in self.context[master].slave.keys()
+            return False   
+        else:
+            return varname in self.context
+
+
+class CreoleClient():
+    def get(self, path):
+        path = path.replace('/', '.')
+        if path.startswith('.'):
+            path = path[1:]
+        if '.' not in path:
+            return self.context[path]
+        else:
+            root, path = path.split('.', 1)
+            obj = self.context[root]
+            for var in path.split('.'):
+                obj = getattr(obj, var)
+            return obj
+
+
+def is_empty(data):
+    if str(data) in ['', '""', "''", "[]", "['']", '[""]', "None"]:
+        return True
+    return False
+
+
+class CheetahTemplate(oriCheetahTemplate):
+    def __init__(self,
+                 filename: str,
+                 source: str,
+                 context,
+                 eosfunc: Dict,
+                 extra_context: Dict,
+                 ):
+        creole_client = CreoleClient()
+        creole_client.context=context
+        extra_context['is_defined'] = IsDefined(context)
+        extra_context['creole_client'] = creole_client
+        extra_context['is_empty'] = is_empty
+        extra_context['_creole_filename'] = extra_context['rougail_filename']
+        super().__init__(filename, source, context, eosfunc, extra_context)
+
+
+# Sync to creole.py
+def process(filename: str,
+            source: str,
+            true_destfilename: str,
+            destfilename: str,
+            destdir: str,
+            variable: Any,
+            rougail_variables_dict: Dict,
+            eosfunc: Dict,
+            ):
+    """Process a cheetah template
+    """
+    # full path of the destination file
+    try:
+        extra_context = {'normalize_family': normalize_family,
+                         'rougail_filename': true_destfilename,
+                         'rougail_destination_dir': destdir,
+                         }
+        if variable is not None:
+            extra_context['rougail_variable'] = variable
+        cheetah_template = CheetahTemplate(filename,
+                                           source,
+                                           rougail_variables_dict,
+                                           eosfunc,
+                                           extra_context,
+                                           )
+        data = str(cheetah_template)
+    except NotFound as err:  # pragma: no cover
+        varname = err.args[0][13:-1]
+        if filename:
+            msg = f"Error: unknown variable used in template {filename} to {destfilename}: {varname}"
+        else:
+            msg = f"Error: unknown variable used in file {destfilename}: {varname}"
+        raise TemplateError(_(msg)) from err
+    except Exception as err:  # pragma: no cover
+        if filename:
+            msg = _(f"Error while instantiating template {filename} to {destfilename}: {err}")
+        else:
+            msg = _(f"Error while instantiating filename {destfilename}: {err}")
+        raise TemplateError(msg) from err
+
+    with open(destfilename, 'w') as file_h:
+        file_h.write(data)

src/rougail/template/engine/jinja2.py

@@ -0,0 +1,74 @@
+"""Jinja2 engine
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from typing import Any, Dict
+from jinja2 import Environment, FileSystemLoader
+from jinja2.exceptions import UndefinedError
+
+from ...i18n import _
+from ...utils import normalize_family
+from ...error import TemplateError
+
+
+def process(filename: str,
+            source: str,
+            true_destfilename: str,
+            destfilename: str,
+            destdir: str,
+            variable: Any,
+            rougail_variables_dict: Dict,
+            eosfunc: Dict,
+            ):
+    """Process a cheetah template
+    """
+    # full path of the destination file
+    dir_name, template_name = filename.rsplit('/', 1)
+    if source is not None:  # pragma: no cover
+        raise TemplateError(_('source is not supported for jinja2'))
+        rougail_variables_dict['rougail_variable'] = variable
+    if variable is not None:
+        var = {'rougail_variable': variable}
+    else:
+        var = {}
+    try:
+        # extra_context = {'normalize_family': normalize_family,
+        # eosfunc
+        env = Environment(loader=FileSystemLoader([dir_name, destdir]))
+        template = env.get_template(template_name)
+        data = template.render(**rougail_variables_dict,
+                               rougail_filename=true_destfilename,
+                               rougail_destination_dir=destdir,
+                               **var,
+                               )
+    except UndefinedError as err:  # pragma: no cover
+        varname = err
+        msg = f"Error: unknown variable used in template {filename} to {destfilename}: {varname}"
+        raise TemplateError(_(msg)) from err
+
+    if not data.endswith('\n'):
+        data = data + '\n'
+    with open(destfilename, 'w') as file_h:
+        file_h.write(data)

src/rougail/template/engine/none.py

@@ -0,0 +1,35 @@
+"""None engine
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from typing import Any
+from shutil import copy
+
+
+def process(filename: str,
+            destfilename: str,
+            **kwargs
+            ):
+    copy(filename, destfilename)

src/rougail/template/systemd.py

@@ -0,0 +1,163 @@
+"""Template langage for Rougail to create file and systemd file
+
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+
+from typing import Dict, Any
+from os import makedirs, symlink
+from os.path import dirname, isfile, join
+from ipaddress import ip_network
+
+from .base import RougailBaseTemplate
+from ..i18n import _
+from ..error import FileNotFound
+
+
+ROUGAIL_IP_TEMPLATE = """[Service]
+%for %%ip in %%rougail_variable
+IPAddressAllow=%%ip
+%end for
+IPAddressDeny=any
+"""
+
+
+ROUGAIL_TMPL_TEMPLATE = """%def display(%%file, %%filename)
+%if %%filename.startswith('/etc/') or %%filename.startswith('/var/') or %%filename.startswith('/srv/')
+C %%filename %%file.mode %%file.owner %%file.group - /usr/local/lib%%filename
+z %%filename - - - - -
+%end if
+%end def
+%for %%service in %%services
+  %if %%service.activate is True and %%hasattr(%%service, 'files')
+    %for %%file in %%service.files
+      %if %%file.activate is True and %%file.included != 'content'
+        %if %%isinstance(%%file.name, list)
+          %for %%filename in %%file.name
+%%display(%%file, %%filename)%slurp
+          %end for
+        %else
+%%display(%%file, %%file.name)%slurp
+        %end if
+      %end if
+    %end for
+  %end if
+%end for
+"""
+
+
+class RougailSystemdTemplate(RougailBaseTemplate):
+    def __init__(self,  # pylint: disable=R0913
+                 config: 'Config',
+                 rougailconfig: 'RougailConfig'=None,
+                 ) -> None:
+        self.ip_per_service = None
+        super().__init__(config, rougailconfig)
+
+    def _instance_files(self,
+                        filevar: Dict,
+                        destfile: str,
+                        service_name: str,
+                        variable,
+                        idx: int,
+                        ) -> tuple:
+        source = filevar['source']
+        if not isfile(source):  # pragma: no cover
+            raise FileNotFound(_(f"File {source} does not exist."))
+        tmp_file = join(self.tmp_dir, source)
+        #self.instance_file(fill, 'files')
+        if variable:
+            var = variable[idx]
+        else:
+            var = None
+        return tmp_file, None, destfile, var
+
+    def _instance_overrides(self,
+                            filevar: Dict,
+                            destfile,
+                            service_name: str,
+                            *args,
+                            ) -> tuple:
+        source = filevar['source']
+        if not isfile(source):  # pragma: no cover
+            raise FileNotFound(_(f"File {source} does not exist."))
+        tmp_file = join(self.tmp_dir, source)
+        service_name = filevar['name']
+        return tmp_file, None, f'/systemd/system/{service_name}.service.d/rougail.conf', None
+
+    def _instance_ip(self,
+                     filevar: Dict,
+                     ip,
+                     service_name: str,
+                     var: Any,
+                     idx: int,
+                     *args,
+                     ) -> tuple:
+        if self.ip_per_service is None:
+            self.ip_per_service = []
+        if 'netmask' in filevar:
+            if isinstance(filevar["netmask"], list):
+                netmask = filevar['netmask'][idx]
+            else:
+                netmask = filevar['netmask']
+            self.ip_per_service.append(str(ip_network(f'{ip}/{netmask}')))
+        elif ip:
+            self.ip_per_service.append(ip)
+
+    def desactive_service(self,
+                          service_name: str,
+                          ):
+        filename = f'{self.destinations_dir}/systemd/system/{service_name}.service'
+        makedirs(dirname(filename), exist_ok=True)
+        symlink('/dev/null', filename)
+
+    def post_instance_service(self,
+                              service_name: str,
+                              ) -> None:  # pragma: no cover
+        if self.ip_per_service is None:
+            return
+        destfile = f'/systemd/system/{service_name}.service.d/rougail_ip.conf'
+        destfilename = join(self.destinations_dir, destfile[1:])
+        makedirs(dirname(destfilename), exist_ok=True)
+        self.log.info(_(f"creole processing: '{destfilename}'"))
+        self.engines['creole'].process(filename=None,
+                                       source=ROUGAIL_IP_TEMPLATE,
+                                       true_destfilename=destfile,
+                                       destfilename=destfilename,
+                                       destdir=self.destinations_dir,
+                                       variable=self.ip_per_service,
+                                       rougail_variables_dict=self.rougail_variables_dict,
+                                       eosfunc=self.eosfunc,
+                                       )
+        self.ip_per_service = None
+
+    def post_instance(self):
+        destfile = '/tmpfiles.d/rougail.conf'
+        destfilename = join(self.destinations_dir, destfile[1:])
+        makedirs(dirname(destfilename), exist_ok=True)
+        self.log.info(_(f"creole processing: '{destfilename}'"))
+        self.engines['creole'].process(filename=None,
+                                       source=ROUGAIL_TMPL_TEMPLATE,
+                                       true_destfilename=destfile,
+                                       destfilename=destfilename,
+                                       destdir=self.destinations_dir,
+                                       variable=None,
+                                       rougail_variables_dict=self.rougail_variables_dict,
+                                       eosfunc=self.eosfunc,
+                                       )

src/rougail/tiramisu.py

@@ -1,13 +1,41 @@
+"""Redefine Tiramisu object
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
 try:
     from tiramisu3 import DynOptionDescription
-except:
+except ModuleNotFoundError:
     from tiramisu import DynOptionDescription
 from .utils import normalize_family
 
 
 class ConvertDynOptionDescription(DynOptionDescription):
+    """Suffix could be an integer, we should convert it in str
+    Suffix could also contain invalid character, so we should "normalize" it
+    """
     def convert_suffix_to_path(self, suffix):
         if not isinstance(suffix, str):
             suffix = str(suffix)
-        return normalize_family(suffix,
+        return normalize_family(suffix)
-                                check_name=False)

src/rougail/tiramisureflector.py

@@ -1,494 +1,391 @@
 """loader
 flattened XML specific
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 """
-from .config import Config
+from json import dumps
-from .i18n import _
+from os.path import isfile
-from .error import LoaderError
+
-from .annotator import ERASED_ATTRIBUTES, CONVERT_OPTION
+from .annotator import CONVERT_OPTION
+from .objspace import RootRougailObject
 
 
-FUNC_TO_DICT = ['valid_not_equal']
+class Root():  # pylint: disable=R0903
-FORCE_INFORMATIONS = ['help', 'test', 'separator', 'manage']
+    """Root classes
-ATTRIBUTES_ORDER = ('name', 'doc', 'default', 'multi')
+    """
+    path = '.'
+
+
+class BaseElt:  # pylint: disable=R0903
+    """Base element
+    """
+    name = 'baseoption'
+    doc = 'baseoption'
+    path = '.'
 
 
 class TiramisuReflector:
+    """Convert object to tiramisu representation
+    """
     def __init__(self,
-                 xmlroot,
+                 objectspace,
                  funcs_path,
                  ):
-        self.storage = ElementStorage()
+        self.index = 0
-        self.storage.text = ["import imp",
+        self.text = []
-                             f"func = imp.load_source('func', '{funcs_path}')",
+        if funcs_path and isfile(funcs_path):
+            self.text.extend(["from importlib.machinery import SourceFileLoader",
+                              "from importlib.util import spec_from_loader, module_from_spec",
+                              f"loader = SourceFileLoader('func', '{funcs_path}')",
+                              "spec = spec_from_loader(loader.name, loader)",
+                              "func = module_from_spec(spec)",
+                              "loader.exec_module(func)",
                               "for key, value in dict(locals()).items():",
-                             "    if key != ['imp', 'func']:",
+                              "    if key != ['SourceFileLoader', 'func']:",
-                             "        setattr(func, key, value)",
+                              "        setattr(func, key, value)"])
-                             "try:",
+        self.text.extend(["try:",
                           "    from tiramisu3 import *",
                           "except:",
                           "    from tiramisu import *",
-                             "from rougail.tiramisu import ConvertDynOptionDescription",
+                          ])
-                             ]
+        self.objectspace = objectspace
-        self.make_tiramisu_objects(xmlroot)
+        self.make_tiramisu_objects()
-        # parse object
+        if self.objectspace.has_dyn_option is True:
-        self.storage.get('.').get()
+            self.text.append("from rougail.tiramisu import ConvertDynOptionDescription")
 
-    def make_tiramisu_objects(self,
+    def make_tiramisu_objects(self) -> None:
-                              xmlroot,
+        """make tiramisu objects
-                              ):
+        """
-        family = self.get_root_family()
+        baseelt = BaseElt()
-        for xmlelt in self.reorder_family(xmlroot):
+        self.set_name(baseelt)
-            self.iter_family(xmlelt,
+        basefamily = Family(baseelt,
-                             family,
+                            self.text,
-                             None,
+                            self.objectspace,
                             )
-
+        for elt in self.reorder_family():
-    def get_root_family(self):
+            self.populate_family(basefamily,
-        family = Family(BaseElt('baseoption',
+                                 elt,
-                                'baseoption',
-                                ),
-                        self.storage,
-                        False,
-                        '.',
                                  )
-        return family
+        self.baseelt = baseelt
 
-    def reorder_family(self, xmlroot):
+    def reorder_family(self):
-        # variable_namespace family has to be loaded before any other family
+        """variable_namespace family has to be loaded before any other family
-        # because `extra` family could use `variable_namespace` variables.
+        because `extra` family could use `variable_namespace` variables.
-        if hasattr(xmlroot, 'variables'):
+        """
-            if Config['variable_namespace'] in xmlroot.variables:
+        if hasattr(self.objectspace.space, 'variables'):
-                yield xmlroot.variables[Config['variable_namespace']]
+            variable_namespace = self.objectspace.rougailconfig['variable_namespace']
-            for xmlelt, value in xmlroot.variables.items():
+            if variable_namespace in self.objectspace.space.variables:
-                if xmlelt != Config['variable_namespace']:
+                yield self.objectspace.space.variables[variable_namespace]
+            for elt, value in self.objectspace.space.variables.items():
+                if elt != self.objectspace.rougailconfig['variable_namespace']:
                     yield value
-        if hasattr(xmlroot, 'services'):
+        if hasattr(self.objectspace.space, 'services'):
-            yield xmlroot.services
+            yield self.objectspace.space.services
-
-    def get_attributes(self, space):  # pylint: disable=R0201
-        for attr in dir(space):
-            if not attr.startswith('_') and attr not in ERASED_ATTRIBUTES:
-                yield attr
-
-    def get_children(self,
-                     space,
-                     ):
-        for tag in self.get_attributes(space):
-            children = getattr(space, tag)
-            if children.__class__.__name__ == 'Family':
-                children = [children]
-            if isinstance(children, dict):
-                children = list(children.values())
-            if isinstance(children, list):
-                yield tag, children
-
-    def iter_family(self,
-                    child,
-                    family,
-                    subpath,
-                    ):
-        tag = child.__class__.__name__
-        if tag == 'Variable':
-            function = self.populate_variable
-        elif tag == 'Property':
-            # property already imported with family
-            return
-        else:
-            function = self.populate_family
-        #else:
-        #    raise Exception('unknown tag {}'.format(child.tag))
-        function(family,
-                 child,
-                 subpath,
-                 )
 
     def populate_family(self,
                         parent_family,
                         elt,
-                        subpath,
                         ):
-        path = self.build_path(subpath,
+        """Populate family
-                               elt,
+        """
-                               )
+        self.set_name(elt)
-        tag = elt.__class__.__name__
         family = Family(elt,
-                        self.storage,
+                        self.text,
-                        tag == 'Leadership',
+                        self.objectspace,
-                        path,
                         )
         parent_family.add(family)
-        for tag, children in self.get_children(elt):
+        for children in vars(elt).values():
+            if isinstance(children, self.objectspace.family):
+                self.populate_family(family,
+                                     children,
+                                     )
+                continue
+            if isinstance(children, dict):
+                children = list(children.values())
+            if isinstance(children, list):
                 for child in children:
-                self.iter_family(child,
+                    if isinstance(child, self.objectspace.property_) or \
-                                 family,
+                            not isinstance(child, RootRougailObject):
-                                 path,
+                        continue
-                                 )
+                    if isinstance(child, self.objectspace.variable):
-
+                        self.set_name(child)
-    def populate_variable(self,
+                        family.add(Variable(child,
-                          family,
+                                            self.text,
-                          elt,
+                                            self.objectspace,
-                          subpath,
-                          ):
-        is_follower = False
-        is_leader = False
-        if family.is_leader:
-            if elt.name != family.elt.name:
-                is_follower = True
-            else:
-                is_leader = True
-        family.add(Variable(elt,
-                            self.storage,
-                            is_follower,
-                            is_leader,
-                            self.build_path(subpath,
-                                            elt,
-                                            )
                                             ))
+                    else:
+                        self.populate_family(family,
+                                             child,
+                                             )
 
-    def build_path(self,
+    def set_name(self,
-                   subpath,
                  elt,
                  ):
-        if subpath is None:
+        """Set name
-            return elt.name
+        """
-        return subpath + '.' + elt.name
+        elt.reflector_name = f'option_{self.index}'
-
-    def get_text(self):
-        return '\n'.join(self.storage.get('.').get_text())
-
-
-class BaseElt:
-    def __init__(self,
-                 name,
-                 doc,
-                 ):
-        self.name = name
-        self.doc = doc
-
-    def __iter__(self):
-        return iter([])
-
-
-class ElementStorage:
-    def __init__(self,
-                 ):
-        self.paths = {}
-        self.text = []
-        self.index = 0
-
-    def add(self, path, elt):
-        self.paths[path] = (elt, self.index)
         self.index += 1
 
-    def get(self, path):
+    def get_text(self):
-        if path not in self.paths or self.paths[path][0] is None:
+        """Get text
-            raise LoaderError(_('there is no element for path {}').format(path))
+        """
-        return self.paths[path][0]
+        self.baseelt.reflector_object.get()  # pylint: disable=E1101
-
+        return '\n'.join(self.text)
-    def get_name(self, path):
-        if path not in self.paths:
-            raise LoaderError(_('there is no element for index {}').format(path))
-        return f'option_{self.paths[path][1]}'
 
 
 class Common:
+    """Common function for variable and family
+    """
     def __init__(self,
                  elt,
-                 storage,
+                 text,
-                 is_leader,
+                 objectspace,
-                 path,
                  ):
+        self.elt = elt
         self.option_name = None
-        self.path = path
+        self.text = text
-        self.attrib = {}
+        self.objectspace = objectspace
-        self.informations = {}
+        self.elt.reflector_object = self
-        self.storage = storage
+        self.object_type = None
-        self.is_leader = is_leader
-        self.storage.add(self.path, self)
 
-    def populate_properties(self, child):
+    def get(self):
-        if child.type == 'calculation':
+        """Get tiramisu's object
-            action = f"ParamValue('{child.name}')"
+        """
-            option_name = self.storage.get(child.source).get()
+        if self.option_name is None:
-            kwargs = f"'condition': ParamOption({option_name}, todict=True), 'expected': ParamValue('{child.expected}')"
+            self.option_name = self.elt.reflector_name
+            self.populate_attrib()
+            self.populate_informations()
+        return self.option_name
+
+    def populate_attrib(self):
+        """Populate attributes
+        """
+        keys = {'name': self.convert_str(self.elt.name)}
+        if hasattr(self.elt, 'doc'):
+            keys['doc'] = self.convert_str(self.elt.doc)
+        self._populate_attrib(keys)
+        if hasattr(self.elt, 'properties'):
+            keys['properties'] = self.properties_to_string(self.elt.properties)
+        attrib = ', '.join([f'{key}={value}' for key, value in keys.items()])
+        self.text.append(f'{self.option_name} = {self.object_type}({attrib})')
+
+    def _populate_attrib(self,
+                         keys: dict,
+                         ) -> None:  # pragma: no cover
+        raise NotImplementedError()
+
+    @staticmethod
+    def convert_str(value):
+        """convert string
+        """
+        return dumps(value, ensure_ascii=False)
+
+    def properties_to_string(self,
+                             values: list,
+                             ) -> None:
+        """Change properties to string
+        """
+        properties = [self.convert_str(property_) for property_ in values
+                      if isinstance(property_, str)]
+        calc_properties = [self.calc_properties(property_) for property_ in values \
+                if isinstance(property_, self.objectspace.property_)]
+        return 'frozenset({' + ', '.join(sorted(properties) + calc_properties) + '})'
+
+    def calc_properties(self,
+                        child,
+                        ) -> str:
+        """Populate properties
+        """
+        option_name = child.source.reflector_object.get()
+        kwargs = (f"'condition': ParamOption({option_name}, todict=True, notraisepropertyerror=True), "
+                  f"'expected': {self.populate_param(child.expected)}")
         if child.inverse:
             kwargs += ", 'reverse_condition': ParamValue(True)"
-            prop = 'Calculation(calc_value, Params(' + action + ', kwargs={' + kwargs + '}))'
+        return (f"Calculation(func.calc_value, Params(ParamValue('{child.name}'), "
-        else:
+                f"kwargs={{{kwargs}}}))")
-            prop = "'" + child.text + "'"
-        if self.attrib['properties']:
-            self.attrib['properties'] += ', '
-        self.attrib['properties'] += prop
-
-    def get_attrib(self, attrib):
-        ret_list = []
-        for key, value in self.attrib.items():
-            if value is None:
-                continue
-            if key == 'properties':
-                if not self.attrib[key]:
-                    continue
-                value = "frozenset({" + self.attrib[key] + "})"
-            elif key in ['default', 'multi', 'suffixes', 'validators', 'values']:
-                value = self.attrib[key]
-            elif isinstance(value, str) and key != 'opt' and not value.startswith('['):
-                value = "'" + value.replace("'", "\\\'") + "'"
-            ret_list.append(f'{key}={value}')
-        return ', '.join(ret_list)
 
     def populate_informations(self):
-        for key, value in self.informations.items():
+        """Populate Tiramisu's informations
+        """
+        if not hasattr(self.elt, 'information'):
+            return
+        for key, value in vars(self.elt.information).items():
+            if key == 'xmlfiles':
+                continue
             if isinstance(value, str):
-                value = '"' + value.replace('"', '\"') + '"'
+                value = self.convert_str(value)
-            self.storage.text.append(f'{self.option_name}.impl_set_information("{key}", {value})')
+            self.text.append(f"{self.option_name}.impl_set_information('{key}', {value})")
 
-    def get_text(self,
+    def populate_param(self,
+                       param,
                        ):
-        return self.storage.text
+        """Populate variable parameters
+        """
+        if param.type in ['number', 'boolean', 'nil', 'string', 'port']:
+            value = param.text
+            if param.type == 'string' and value is not None:
+                value = self.convert_str(value)
+            return f'ParamValue({value})'
+        if param.type == 'variable':
+            return self.build_option_param(param)
+        if param.type == 'information':
+            return f'ParamInformation("{param.text}", None)'
+        if param.type == 'target_information':
+            return f'ParamSelfInformation("{param.text}", None)'
+        if param.type == 'suffix':
+            return 'ParamSuffix()'
+        if param.type == 'index':
+            return 'ParamIndex()'
+        raise Exception(f'unknown type {param.type}')  # pragma: no cover
 
-    def get_attributes(self, space):  # pylint: disable=R0201
+    @staticmethod
-        attributes = dir(space)
+    def build_option_param(param,
-        for attr in ATTRIBUTES_ORDER:
+                           ) -> str:
-            if attr in attributes:
+        """build variable parameters
-                yield attr
+        """
-        for attr in dir(space):
+        option_name = param.text.reflector_object.get()
-            if attr not in ATTRIBUTES_ORDER:
+        params = [f'{option_name}']
-                if not attr.startswith('_') and attr not in ERASED_ATTRIBUTES:
+        if hasattr(param, 'suffix'):
-                    value = getattr(space, attr)
+            param_type = 'ParamDynOption'
-                    if not isinstance(value, (list, dict)) and not value.__class__.__name__ == 'Family':
+            family = param.family.reflector_object.get()
-                        yield attr
+            params.extend([f"'{param.suffix}'", f'{family}'])
-
+        else:
-    def get_children(self,
+            param_type = 'ParamOption'
-                     space,
+        if not param.propertyerror:
-                     ):
+            params.append('notraisepropertyerror=True')
-        for attr in dir(space):
+        return "{}({})".format(param_type, ', '.join(params))
-            if not attr.startswith('_') and attr not in ERASED_ATTRIBUTES:
-                value = getattr(space, attr)
-                if isinstance(value, (list, dict)):
-                    children = getattr(space, attr)
-                    if children.__class__.__name__ == 'Family':
-                        children = [children]
-                    if isinstance(children, dict):
-                        children = list(children.values())
-                    if children and isinstance(children, list):
-                        yield attr, children
 
 
 class Variable(Common):
+    """Manage variable
+    """
     def __init__(self,
                  elt,
-                 storage,
+                 text,
-                 is_follower,
+                 objectspace,
-                 is_leader,
-                 path,
                  ):
-        super().__init__(elt,
+        super().__init__(elt, text, objectspace)
-                         storage,
+        self.object_type = CONVERT_OPTION[elt.type]['opttype']
-                         is_leader,
-                         path,
-                         )
-        self.is_follower = is_follower
-        convert_option = CONVERT_OPTION[elt.type]
-        del elt.type
-        self.object_type = convert_option['opttype']
-        self.attrib.update(convert_option.get('initkwargs', {}))
-        if self.object_type != 'SymLinkOption':
-            self.attrib['properties'] = []
-            self.attrib['validators'] = []
-        self.elt = elt
 
-    def get(self):
+    def _populate_attrib(self,
-        if self.option_name is None:
+                         keys: dict,
-            self.populate_attrib()
+                         ):
-            if self.object_type == 'SymLinkOption':
+        if hasattr(self.elt, 'opt'):
-                self.attrib['opt'] = self.storage.get(self.attrib['opt']).get()
+            keys['opt'] = self.elt.opt.reflector_object.get()
+        if hasattr(self.elt, 'values'):
+            values = self.elt.values
+            if values[0].type == 'variable':
+                value = values[0].name.reflector_object.get()
+                keys['values'] = f"Calculation(func.calc_value, Params((ParamOption({value}))))"
+            elif values[0].type == 'function':
+                keys['values'] = self.calculation_value(self.elt.values[0], [])
             else:
-                self.parse_children()
+                keys['values'] = str(tuple([val.name for val in values]))
-            attrib = self.get_attrib(self.attrib)
+        if hasattr(self.elt, 'multi') and self.elt.multi:
-            self.option_name = self.storage.get_name(self.path)
+            keys['multi'] = self.elt.multi
-            self.storage.text.append(f'{self.option_name} = {self.object_type}({attrib})')
+        for key in ['default', 'default_multi']:
-            self.populate_informations()
+            if hasattr(self.elt, key) and getattr(self.elt, key) is not None:
-        return self.option_name
-
-    def populate_attrib(self):
-        for key in self.get_attributes(self.elt):
                 value = getattr(self.elt, key)
-            if key in FORCE_INFORMATIONS:
+                if isinstance(value, str):
-                if key == 'test':
+                    value = self.convert_str(value)
-                    value = value.split(',')
+                elif isinstance(value, self.objectspace.value):
-                    if self.object_type == 'IntOption':
+                    value = self.calculation_value(value, [], calc_multi=value.calc_multi)
-                        value = [int(v) for v in value]
+                keys[key] = value
-                    elif self.object_type == 'FloatOption':
+        if hasattr(self.elt, 'validators'):
-                        value = [float(v) for v in value]
+            keys['validators'] = '[' + ', '.join([self.calculation_value(val,
-                self.informations[key] = value
+                                    ['ParamSelfOption(whole=False)']) for val in self.elt.validators]) + ']'
-            else:
+        for key in ['min_number', 'max_number']:
-                self.attrib[key] = value
+            if hasattr(self.elt, key):
+                keys[key] = getattr(self.elt, key)
+        for key, value in CONVERT_OPTION[self.elt.type].get('initkwargs', {}).items():
+            if isinstance(value, str):
+                value = f"'{value}'"
+            keys[key] = value
 
-    def parse_children(self):
+    def calculation_value(self,
-        if not 'default' in self.attrib or self.attrib['multi']:
+                          child,
-            self.attrib['default'] = []
+                          args,
-        if self.attrib['multi'] == 'submulti' and self.is_follower:
+                          calc_multi=False,
-            self.attrib['default_multi'] = []
+                          ) -> str:
-        choices = []
+        """Generate calculated value
-        if 'properties' in self.attrib:
+        """
-            if self.attrib['properties']:
-                self.attrib['properties'] = "'" + "', '".join(sorted(list(self.attrib['properties']))) + "'"
-            else:
-                self.attrib['properties'] = ''
-        for tag, children in self.get_children(self.elt):
-            for child in children:
-                if tag == 'property':
-                    self.populate_properties(child)
-                elif tag == 'value':
-                    if child.type == 'calculation':
-                        self.attrib['default'] = self.calculation_value(child, [])
-                    else:
-                        self.populate_value(child)
-                elif tag == 'check':
-                    self.attrib['validators'].append(self.calculation_value(child, ['ParamSelfOption()']))
-                elif tag == 'choice':
-                    if child.type == 'calculation':
-                        value = self.storage.get(child.name).get()
-                        choices = f"Calculation(func.calc_value, Params((ParamOption({value}))))"
-                    else:
-                        choices.append(child.name)
-        if choices:
-            if isinstance(choices, list):
-                self.attrib['values'] = str(tuple(choices))
-            else:
-                self.attrib['values'] = choices
-        if self.attrib['default'] == []:
-            del self.attrib['default']
-        elif not self.attrib['multi'] and isinstance(self.attrib['default'], list):
-            self.attrib['default'] = self.attrib['default'][-1]
-        if self.attrib['validators'] == []:
-            del self.attrib['validators']
-        else:
-            self.attrib['validators'] = '[' + ', '.join(self.attrib['validators']) + ']'
-
-    def calculation_value(self, child, args):
         kwargs = []
-        if hasattr(child, 'name'):
         # has parameters
         function = child.name
+        new_args = []
         if hasattr(child, 'param'):
             for param in child.param:
-                    value = self.populate_param(function, param)
+                value = self.populate_param(param)
                 if not hasattr(param, 'name'):
-                        args.append(str(value))
+                    new_args.append(str(value))
                 else:
                     kwargs.append(f"'{param.name}': " + value)
+        if function == 'valid_network_netmask':
+            new_args.extend(args)
         else:
-            # function without any parameter
+            args.extend(new_args)
-            function = child.text.strip()
+            new_args = args
-        ret = f"Calculation(func.{function}, Params((" + ', '.join(args) + "), kwargs=" + "{" + ', '.join(kwargs) + "})"
+        ret = f'Calculation(func.{function}, Params((' + ', '.join(new_args) + ')'
+        if kwargs:
+            ret += ', kwargs={' + ', '.join(kwargs) + '}'
+        ret += ')'
         if hasattr(child, 'warnings_only'):
             ret += f', warnings_only={child.warnings_only}'
-        return ret + ')'
+        ret = ret + ')'
-
+        if calc_multi:
-    def populate_param(self,
+            ret = '[' + ret + ']'
-                       function: str,
+        return ret
-                       param,
-                       ):
-        if param.type == 'string':
-            return f'ParamValue("{param.text}")'
-        elif param.type == 'number':
-            return f'ParamValue({param.text})'
-        elif param.type == 'variable':
-            value = {'option': param.text,
-                     'notraisepropertyerror': param.notraisepropertyerror,
-                     'todict': function in FUNC_TO_DICT,
-                     }
-            if hasattr(param, 'suffix'):
-                value['suffix'] = param.suffix
-            return self.build_param(value)
-        elif param.type == 'information':
-            return f'ParamInformation("{param.text}", None)'
-        elif param.type == 'suffix':
-            return f'ParamSuffix()'
-        raise LoaderError(_('unknown param type {}').format(param.type))
-
-    def populate_value(self,
-                       child,
-                       ):
-        value = child.name
-        if self.attrib['multi'] == 'submulti':
-            self.attrib['default_multi'].append(value)
-        elif self.is_follower:
-            self.attrib['default_multi'] = value
-        elif self.attrib['multi']:
-            self.attrib['default'].append(value)
-            if not self.is_leader:
-                self.attrib['default_multi'] = value
-        elif isinstance(value, (int, float)):
-            self.attrib['default'].append(value)
-        else:
-            self.attrib['default'].append("'" + value + "'")
-
-    def build_param(self,
-                    param,
-                    ):
-        option_name = self.storage.get(param['option']).get()
-        if 'suffix' in param:
-            family = '.'.join(param['option'].split('.')[:-1])
-            family_option = self.storage.get_name(family)
-            return f"ParamDynOption({option_name}, '{param['suffix']}', {family_option}, notraisepropertyerror={param['notraisepropertyerror']}, todict={param['todict']})"
-        return f"ParamOption({option_name}, notraisepropertyerror={param['notraisepropertyerror']}, todict={param['todict']})"
 
 
 class Family(Common):
+    """Manage family
+    """
     def __init__(self,
                  elt,
-                 storage,
+                 text,
-                 is_leader,
+                 objectspace,
-                 path,
                  ):
-        super().__init__(elt,
+        super().__init__(elt, text, objectspace)
-                         storage,
+        if hasattr(self.elt, 'suffixes'):
-                         is_leader,
+            self.objectspace.has_dyn_option = True
-                         path,
+            self.object_type = 'ConvertDynOptionDescription'
-                         )
+        elif isinstance(self.elt, self.objectspace.leadership):
+            self.object_type = 'Leadership'
+        else:
+            self.object_type = 'OptionDescription'
         self.children = []
-        self.elt = elt
 
     def add(self, child):
+        """Add a child
+        """
         self.children.append(child)
 
-    def get(self):
+    def _populate_attrib(self,
-        if not self.option_name:
+                         keys: list,
-            self.populate_attrib()
+                         ) -> None:
-            self.parse_children()
+        if hasattr(self.elt, 'suffixes'):
-            self.option_name = self.storage.get_name(self.path)
+            dyn = self.elt.suffixes.reflector_object.get()
-            object_name = self.get_object_name()
+            keys['suffixes'] = f"Calculation(func.calc_value, Params((ParamOption({dyn}, notraisepropertyerror=True))))"
-            attrib = self.get_attrib(self.attrib) + ', children=[' + ', '.join([child.get() for child in self.children]) + ']'
+        keys['children'] = '[' + ', '.join([child.get() for child in self.children]) + ']'
-            self.storage.text.append(f'{self.option_name} = {object_name}({attrib})')
-            self.populate_informations()
-        return self.option_name
-
-    def populate_attrib(self):
-        for key in self.get_attributes(self.elt):
-            value = getattr(self.elt, key)
-            if key in FORCE_INFORMATIONS:
-                self.informations[key] = value
-            elif key == 'dynamic':
-                dynamic = self.storage.get(value).get()
-                self.attrib['suffixes'] = f"Calculation(func.calc_value, Params((ParamOption({dynamic}))))"
-            else:
-                self.attrib[key] = value
-
-    def parse_children(self):
-        if 'properties' in self.attrib:
-            self.attrib['properties'] = "'" + "', '".join(sorted(list(self.attrib['properties']))) + "'"
-        if hasattr(self.elt, 'property'):
-            #self.attrib['properties'] = ''
-            for child in self.elt.property:
-                self.populate_properties(child)
-            if not self.attrib['properties']:
-                del self.attrib['properties']
-
-    def get_object_name(self):
-        if 'suffixes' in self.attrib:
-            return 'ConvertDynOptionDescription'
-        elif not self.is_leader:
-            return 'OptionDescription'
-        return 'Leadership'

src/rougail/update.py

@@ -0,0 +1,573 @@
+"""Update Rougail XML file to new version
+
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+
+from typing import List
+from os.path import join, isfile, basename
+from os import listdir
+from lxml.etree import DTD, parse, XMLParser, XMLSyntaxError  # pylint: disable=E0611
+from lxml.etree import Element, SubElement, tostring
+from ast import parse as ast_parse
+
+from .i18n import _
+from .error import UpgradeError
+
+from .utils import normalize_family
+
+
+VERSIONS = {'creole': ['1'],
+            'rougail': ['0.9'],
+            }
+
+
+def get_function_name(root, version):
+    version = version.replace('.', '_')
+    return f'update_{root}_{version}'
+
+
+FUNCTION_VERSIONS = [(root, version, get_function_name(root, version)) for root, versions in VERSIONS.items() for version in versions]
+
+
+class RougailUpgrade:
+    def __init__(self, test=False, upgrade_help=None):
+        self.test = test
+        if upgrade_help is None:
+            upgrade_help = {}
+        self.upgrade_help = upgrade_help
+
+    def load_xml_from_folders(self,
+                              srcfolder: str,
+                              dstfolder: str,
+                              namespace: str,
+                              ):
+        """Loads all the XML files located in the xmlfolders' list
+
+        :param xmlfolders: list of full folder's name
+        """
+        filenames = [filename for filename in listdir(srcfolder) if filename.endswith('.xml')]
+        filenames.sort()
+        for filename in filenames:
+            xmlsrc = join(srcfolder, filename)
+            xmldst = join(dstfolder, filename)
+            try:
+                parser = XMLParser(remove_blank_text=True)
+                document = parse(xmlsrc, parser)
+            except XMLSyntaxError as err:
+                raise Exception(_(f'not a XML file: {err}')) from err
+            root = document.getroot()
+            search_function_name = get_function_name(root.tag, root.attrib.get('version', '1'))
+            function_found = False
+            for root_name, version, function_version in FUNCTION_VERSIONS:
+                if function_found and hasattr(self, function_version):
+                    print(f'  - convert {filename} to version {version}')
+                    upgrade_help = self.upgrade_help.get(function_version, {}).get(filename, {})
+                    if upgrade_help.get('remove') is True:
+                        continue
+                    root = getattr(self, function_version)(root, upgrade_help, namespace)
+                if function_version == search_function_name:
+                    function_found = True
+            with open(xmldst, 'wb') as xmlfh:
+                xmlfh.write(tostring(root, pretty_print=True, encoding="UTF-8", xml_declaration=True))
+    #            if
+    #            if not self.dtd.validate(document):
+    #                dtd_error = self.dtd.error_log.filter_from_errors()[0]
+    #                msg = _(f'not a valid XML file: {dtd_error}')
+    #                raise DictConsistencyError(msg, 43, [xmlfile])
+    #            yield xmlfile, document.getroot()
+
+
+    def update_rougail_0_9(self,
+                           root: 'Element',
+                           upgrade_help: dict,
+                           namespace: str,
+                           ) -> 'Element':
+        # rename root
+        root.tag = 'rougail'
+        root.attrib['version'] = '0.9'
+        variables_auto_valid_enum = {}
+        variables_help = {}
+        families_help = {}
+        variables = {}
+        families = {}
+        valid_enums = {}
+        current_service = upgrade_help.get('services', {}).get('default', 'unknown')
+        files = {current_service: {}}
+        ips = {current_service: []}
+        servicelists = {}
+        test_unknowns_variables = set()
+        autos = []
+        constraints_obj = None
+        for subelement in root:
+            if not isinstance(subelement.tag, str):
+                # XML comment
+                continue
+            if subelement.tag == 'family_action':
+                root.remove(subelement)
+                continue
+            for subsubelement in subelement:
+                if not isinstance(subsubelement.tag, str):
+                    # XML comment
+                    continue
+                for subsubsubelement in subsubelement:
+                    if not isinstance(subsubsubelement.tag, str):
+                        # XML comment
+                        continue
+                    if subsubsubelement.tag == 'variable':
+                        if subsubsubelement.attrib['name'] in upgrade_help.get('variables', {}).get('remove', []):
+                            subsubelement.remove(subsubsubelement)
+                            continue
+                        if subsubsubelement.attrib['name'] in upgrade_help.get('variables', {}).get('type', {}):
+                            subsubsubelement.attrib['type'] = upgrade_help['variables']['type'][subsubsubelement.attrib['name']]
+                        if subsubsubelement.attrib['name'] in upgrade_help.get('variables', {}).get('hidden', {}).get('add', []):
+                            subsubsubelement.attrib['hidden'] = 'True'
+                        if subsubsubelement.attrib['name'] in upgrade_help.get('variables', {}).get('hidden', {}).get('remove', []):
+                            self.remove(subsubsubelement, 'hidden', optional=True)
+                        if subsubsubelement.attrib['name'] in upgrade_help.get('variables', {}).get('mandatory', {}).get('remove', []):
+                            self.remove(subsubsubelement, 'mandatory')
+                        if subsubsubelement.attrib['name'] in upgrade_help.get('variables', {}).get('mandatory', {}).get('add', []):
+                            subsubsubelement.attrib['mandatory'] = 'True'
+                        if subsubsubelement.attrib['name'] in upgrade_help.get('variables', {}).get('type', {}):
+                            subsubsubelement.attrib['type'] = upgrade_help.get('variables', {}).get('type', {})[subsubsubelement.attrib['name']]
+                        if namespace == 'configuration':
+                            path = subsubsubelement.attrib['name']
+                            npath = normalize_family(subsubsubelement.attrib['name'])
+                        else:
+                            path = namespace + '.' + subsubelement.attrib['name'] + '.' + subsubsubelement.attrib['name']
+                            npath = normalize_family(namespace) + '.' + normalize_family(subsubelement.attrib['name']) + '.' + normalize_family(subsubsubelement.attrib['name'])
+                        variables[path] = subsubsubelement
+                        variables[npath] = subsubsubelement
+                        type = subsubsubelement.attrib.get('type')
+                        if type in ['oui/non', 'yes/no', 'on/off']:
+                            variables_auto_valid_enum.setdefault(subsubsubelement.attrib['type'], []).append(path)
+                            del subsubsubelement.attrib['type']
+                        elif type == 'hostname_strict':
+                            subsubsubelement.attrib['type'] = 'hostname'
+                        elif type in ['domain', 'domain_strict']:
+                            subsubsubelement.attrib['type'] = 'domainname'
+                        elif type == 'string':
+                            del subsubsubelement.attrib['type']
+                        if self.test and subsubsubelement.attrib.get('auto_freeze') == 'True':
+                            del subsubsubelement.attrib['auto_freeze']
+                        #if self.test and subsubsubelement.attrib.get('redefine') == 'True':
+                        #    subsubsubelement.attrib['exists'] = 'False'
+                        #    del subsubsubelement.attrib['redefine']
+                        if subsubsubelement.attrib['name'] in upgrade_help.get('variables', {}).get('redefine', {}).get('remove', {}):
+                            del subsubsubelement.attrib['redefine']
+                        if subsubsubelement.attrib['name'] in upgrade_help.get('variables', {}).get('remove_check', []):
+                            subsubsubelement.attrib['remove_check'] = 'True'
+                        if subsubsubelement.attrib['name'] in upgrade_help.get('variables', {}).get('mode', {}).get('modify', {}):
+                            subsubsubelement.attrib['mode'] = upgrade_help.get('variables', {}).get('mode', {}).get('modify', {})[subsubsubelement.attrib['name']]
+                        if subsubsubelement.attrib['name'] in upgrade_help.get('variables', {}).get('type', {}).get('modify', {}):
+                            subsubsubelement.attrib['type'] = upgrade_help.get('variables', {}).get('type', {}).get('modify', {})[subsubsubelement.attrib['name']]
+                            type = subsubsubelement.attrib['type']
+                        if subsubsubelement.attrib['name'] in upgrade_help.get('variables', {}).get('test', {}):
+                            subsubsubelement.attrib['test'] = upgrade_help.get('variables', {}).get('test', {})[subsubsubelement.attrib['name']]
+                        if subsubsubelement.attrib['name'] in upgrade_help.get('variables', {}).get('remove_value', []):
+                            for value in subsubsubelement:
+                                subsubsubelement.remove(value)
+                        if subsubsubelement.attrib['name'] != normalize_family(subsubsubelement.attrib['name']):
+                            if "description" not in subsubsubelement.attrib:
+                                subsubsubelement.attrib['description'] = subsubsubelement.attrib['name']
+                            subsubsubelement.attrib['name'] = normalize_family(subsubsubelement.attrib['name'])
+                    elif subsubsubelement.tag == 'param':
+                        if subsubelement.tag == 'check' and subsubelement.attrib['target'] in upgrade_help.get('check', {}).get('remove', []):
+                            continue
+                        type = subsubsubelement.attrib.get('type')
+                        if type == 'eole':
+                            subsubsubelement.attrib['type'] = 'variable'
+                            type = 'variable'
+                        if type == 'python':
+                            subsubsubelement.attrib['type'] = 'function'
+                            if subsubsubelement.text.startswith('range('):
+                                func_ast = ast_parse(subsubsubelement.text)
+                                subsubsubelement.text = 'range'
+                                for arg in func_ast.body[0].value.args:
+                                    SubElement(subsubelement, 'param', type='number').text = str(arg.value)
+                            else:
+                                raise Exception(f'{subsubsubelement.text} is not a supported function')
+                            type = 'function'
+                        elif type in ('container', 'context'):
+                            raise UpgradeError(_(f'cannot convert param with type "{type}"'))
+                        if subsubelement.attrib['name'] == 'valid_entier' and not 'type' in subsubsubelement.attrib:
+                            subsubsubelement.attrib['type'] = 'number'
+                        if (not type or type == 'variable') and subsubsubelement.text in upgrade_help.get('variables', {}).get('remove', []):
+                            subsubelement.remove(subsubsubelement)
+                            continue
+                        if subsubelement.attrib['name'] == 'valid_enum' and not type:
+                            if subsubsubelement.attrib.get('name') == 'checkval':
+                                if subsubelement.attrib['target'] in upgrade_help.get('check', {}).get('valid_enums', {}).get('checkval', {}).get('remove', []):
+                                    subsubelement.remove(subsubsubelement)
+                                    continue
+                                raise UpgradeError(_('checkval in valid_enum is no more supported'))
+                            if subsubsubelement.text.startswith('['):
+                                for val in eval(subsubsubelement.text):
+                                    SubElement(subsubelement, 'param').text = str(val)
+                                subsubelement.remove(subsubsubelement)
+                        self.move(subsubsubelement, 'hidden', 'propertyerror', optional=True)
+                        if type == 'variable' and subsubsubelement.text in upgrade_help.get('variables', {}).get('rename', []):
+                            subsubsubelement.text = upgrade_help.get('variables', {}).get('rename', [])[subsubsubelement.text]
+                        up = upgrade_help.get('params', {}).get(subsubsubelement.text)
+                        if up:
+                            if 'text' in up:
+                                subsubsubelement.text = up['text']
+                            if 'type' in up:
+                                subsubsubelement.attrib['type'] = up['type']
+                    elif subsubsubelement.tag == 'target':
+                        type = subsubsubelement.attrib.get('type')
+                        if type in ['service_accesslist', 'service_restrictionlist', 'interfacelist', 'hostlist']:
+                            subsubelement.remove(subsubsubelement)
+                        elif (not type or type == 'variable') and subsubsubelement.text in upgrade_help.get('variables', {}).get('remove', []):
+                            subsubelement.remove(subsubsubelement)
+                        elif type == 'family' and subsubsubelement.text in upgrade_help.get('families', {}).get('remove', []):
+                            subsubelement.remove(subsubsubelement)
+                        elif type == 'actionlist':
+#                            for family in root.find('variables'):
+#                                family_list = SubElement(subsubelement, 'target')
+#                                family_list.attrib['type'] = 'family'
+#                                family_list.text = namespace + '.' + family.attrib['name']
+                            subsubelement.remove(subsubsubelement)
+                        if upgrade_help.get('targets', {}).get(subsubsubelement.text, {}).get('optional'):
+                            subsubsubelement.attrib['optional'] = upgrade_help.get('targets', {}).get(subsubsubelement.text, {}).get('optional')
+                        has_target = False
+                        for target in subsubelement:
+                            if target.tag == 'target':
+                                has_target = True
+                                break
+                        if not has_target:
+                            subelement.remove(subsubelement)
+                        continue
+                    elif subsubsubelement.tag == 'slave':
+                        if subsubsubelement.text in upgrade_help.get('variables', {}).get('remove', []):
+                            subsubelement.remove(subsubsubelement)
+                            continue
+                        subsubsubelement.tag = 'follower'
+                        subsubsubelement.text = normalize_family(subsubsubelement.text)
+                    if subelement.tag == 'containers':
+                        current_service = self.upgrade_container(subsubsubelement, current_service, files, ips, servicelists, upgrade_help)
+                        subsubelement.remove(subsubsubelement)
+                if subelement.tag == 'help':
+                    if subsubelement.tag == 'variable':
+                        if not subsubelement.attrib['name'] in upgrade_help.get('variables', {}).get('remove', []):
+                            variables_help[subsubelement.attrib['name']] = subsubelement.text
+                    elif subsubelement.tag == 'family':
+                        if subsubelement.attrib['name'] not in upgrade_help.get('families', {}).get('remove', []):
+                            families_help[subsubelement.attrib['name']] = subsubelement.text
+                    else:
+                        raise Exception(f'unknown help tag {subsubelement.tag}')
+                    subelement.remove(subsubelement)
+                    continue
+                if subsubelement.tag == 'auto':
+                    if subsubelement.attrib['target'] in upgrade_help.get('variables', {}).get('remove', []):
+                        subelement.remove(subsubelement)
+                        continue
+                    autos.append(subsubelement.attrib['target'])
+                    subsubelement.tag = 'fill'
+                if subsubelement.tag in ['fill', 'check']:
+                    if subsubelement.tag == 'check' and subsubelement.attrib['name'] == 'valid_enum':
+                        if subsubelement.attrib['target'] in upgrade_help.get('variables', {}).get('remove', []):
+                            subelement.remove(subsubelement)
+                            continue
+                        params = []
+                        for param in subsubelement:
+                            if param.tag == 'param':
+                                params.append(param.text)
+                        key = '~'.join(params)
+                        if key in valid_enums:
+                            SubElement(valid_enums[key], 'target').text = subsubelement.attrib['target']
+                            subelement.remove(subsubelement)
+                            continue
+                        valid_enums['~'.join(params)] = subsubelement
+                    if subsubelement.tag == 'fill' and subsubelement.attrib['target'] in upgrade_help.get('fills', {}).get('remove', []):
+                        subelement.remove(subsubelement)
+                        continue
+                    if subsubelement.tag == 'check' and subsubelement.attrib['target'] in upgrade_help.get('check', {}).get('remove', []):
+                        subelement.remove(subsubelement)
+                        continue
+                    if subsubelement.attrib['target'] in upgrade_help.get('variables', {}).get('remove', []):
+                        subelement.remove(subsubelement)
+                        continue
+                    if subsubelement.attrib['name'] == 'valid_networknetmask':
+                        subsubelement.attrib['name'] = 'valid_network_netmask'
+                    target = SubElement(subsubelement, 'target')
+                    target.text = subsubelement.attrib['target']
+                    if self.test:
+                        target.attrib['optional'] = 'True'
+                    del subsubelement.attrib['target']
+                elif subsubelement.tag == 'separators':
+                    subelement.remove(subsubelement)
+                elif subsubelement.tag == 'condition':
+                    if subsubelement.attrib['source'] in upgrade_help.get('variables', {}).get('remove', []):
+                        try:
+                            subelement.remove(subsubelement)
+                        except:
+                            pass
+                    else:
+                        if subsubelement.attrib['name'] == 'hidden_if_not_in':
+                            subsubelement.attrib['name'] = 'disabled_if_not_in'
+                        if subsubelement.attrib['name'] == 'hidden_if_in':
+                            subsubelement.attrib['name'] = 'disabled_if_in'
+                        if subsubelement.attrib['name'] == 'frozen_if_in':
+                            subsubelement.attrib['name'] = 'hidden_if_in'
+                        self.move(subsubelement, 'fallback', 'optional', optional='True')
+                    if subsubelement.attrib['source'] in upgrade_help.get('variables', {}).get('rename', []):
+                        subsubelement.attrib['source'] = upgrade_help.get('variables', {}).get('rename', [])[subsubelement.attrib['source']]
+                elif subsubelement.tag == 'group':
+                    if subsubelement.attrib['master'] in upgrade_help.get('groups', {}).get('remove', []):
+                        subelement.remove(subsubelement)
+                    else:
+                        self.move(subsubelement, 'master', 'leader')
+                        for follower in subsubelement:
+                            if '.' in subsubelement.attrib['leader']:
+                                path = subsubelement.attrib['leader'].rsplit('.', 1)[0] + '.' + follower.text
+                            else:
+                                path = follower.text
+                            self.remove(variables[path], 'multi', optional=True)
+                        if subsubelement.attrib['leader'] in upgrade_help.get('groups', {}).get('reorder', {}):
+                            for follower in subsubelement:
+                                subsubelement.remove(follower)
+                            for follower in upgrade_help.get('groups', {}).get('reorder', {})[subsubelement.attrib['leader']]:
+                                SubElement(subsubelement, 'follower').text = follower
+                if subelement.tag == 'files':
+                    current_service = self.upgrade_container(subsubelement, current_service, files, ips, servicelists, upgrade_help)
+                    subelement.remove(subsubelement)
+                if subsubelement.tag == 'family':
+                    self.remove(subsubelement, 'icon', optional=True)
+                    if subsubelement.attrib['name'] in upgrade_help.get('families', {}).get('mode', {}).get('remove', []) and 'mode' in subsubelement.attrib:
+                        del subsubelement.attrib['mode']
+                    if subsubelement.attrib['name'] in upgrade_help.get('families', {}).get('rename', {}):
+                        subsubelement.attrib['name'] = upgrade_help.get('families', {}).get('rename', {})[subsubelement.attrib['name']]
+                    if subsubelement.attrib['name'] in upgrade_help.get('families', {}).get('remove', []):
+                        subelement.remove(subsubelement)
+                        continue
+                    if subsubelement.attrib['name'] in upgrade_help.get('families', {}).get('hidden', {}).get('add', []):
+                        subsubelement.attrib['hidden'] = 'True'
+                    if subsubelement.attrib['name'] != normalize_family(subsubelement.attrib['name']):
+                        if "description" not in subsubelement.attrib:
+                            subsubelement.attrib['description'] = subsubelement.attrib['name']
+                        subsubelement.attrib['name'] = normalize_family(subsubelement.attrib['name'])
+                    families[subsubelement.attrib['name']] = subsubelement
+            # if empty, remove
+            if is_empty(subelement) or subelement.tag in ['containers', 'files', 'help']:
+                root.remove(subelement)
+                continue
+            # copy subelement
+            if subelement.tag == 'constraints':
+                constraints_obj = subelement
+        services = None
+        service_elt = {}
+        for variable, obj in upgrade_help.get('variables', {}).get('add', {}).items():
+            family = next(iter(families.values()))
+            variables[variable] = SubElement(family, 'variable', name=variable)
+            if 'value' in obj:
+                SubElement(variables[variable], 'value').text = obj['value']
+            for name in ['hidden', 'exists']:
+                if name in obj:
+                    variables[variable].attrib[name] = obj[name]
+        files_is_empty = True
+        for lst in files.values():
+            if len(lst):
+                files_is_empty = False
+                break
+        if not files_is_empty:
+            services = Element('services')
+            root.insert(0, services)
+            for service_name, lst in files.items():
+                service = self.create_service(services, service_name, service_elt, servicelists, upgrade_help)
+                for file_ in lst.values():
+                    self.move(file_, 'name_type', 'file_type', optional=True)
+                    if 'file_type' in file_.attrib and file_.attrib['file_type'] == 'SymLinkOption':
+                        file_.attrib['file_type'] = 'variable'
+                        if variables[file_.text].attrib['type'] == 'string':
+                            variables[file_.text].attrib['type'] = 'filename'
+                    self.remove(file_, 'rm', optional=True)
+                    self.remove(file_, 'mkdir', optional=True)
+                    service.append(file_)
+        ips_is_empty = True
+        for lst in ips.values():
+            if len(lst):
+                ips_is_empty = False
+                break
+        if not ips_is_empty:
+            if services is None:
+                services = Element('services')
+                root.insert(0, services)
+            for service_name, lst in ips.items():
+                service = self.create_service(services, service_name, service_elt, servicelists, upgrade_help)
+                for ip in lst:
+                    if ip.text in upgrade_help.get('ips', {}).get('remove', []):
+                        continue
+                    for type in ['ip_type', 'netmask_type']:
+                        if type in ip.attrib and ip.attrib[type] == 'SymLinkOption':
+                            ip.attrib[type] = 'variable'
+                    if 'netmask' in ip.attrib:
+                        if ip.attrib['netmask'] == '255.255.255.255':
+                            del ip.attrib['netmask']
+                            if ip.text in variables and 'type' not in variables[ip.text].attrib:
+                                variables[ip.text].attrib['type'] = 'ip'
+                        else:
+                            if ip.text in variables and 'type' not in variables[ip.text].attrib:
+                                variables[ip.text].attrib['type'] = 'network'
+                            if ip.attrib['netmask'] in variables and 'type' not in variables[ip.attrib['netmask']].attrib:
+                                variables[ip.attrib['netmask']].attrib['type'] = 'netmask'
+                    elif ip.text in variables and 'type' not in variables[ip.text].attrib:
+                        variables[ip.text].attrib['type'] = 'ip'
+                    self.remove(ip, 'interface')
+                    self.remove(ip, 'interface_type', optional=True)
+                    self.remove(ip, 'service_restrictionlist', optional=True)
+                    service.append(ip)
+        if ips_is_empty and files_is_empty:
+            for service_name in files:
+                if services is None:
+                    services = Element('services')
+                    root.insert(0, services)
+                if service_name != 'unknown':
+                    self.create_service(services, service_name, service_elt, servicelists, upgrade_help)
+        if constraints_obj is None:
+            constraints_obj = SubElement(root, 'constraints')
+        if variables_auto_valid_enum:
+            for type, variables_valid_enum in variables_auto_valid_enum.items():
+                valid_enum = SubElement(constraints_obj, 'check')
+                valid_enum.attrib['name'] = 'valid_enum'
+                for val in type.split('/'):
+                    param = SubElement(valid_enum, 'param')
+                    param.text = val
+                for variable in variables_valid_enum:
+                    target = SubElement(valid_enum, 'target')
+                    target.text = variable
+        for name, text in variables_help.items():
+            variables[name].attrib['help'] = text
+        for name, text in families_help.items():
+            if name in upgrade_help.get('families', {}).get('rename', {}):
+                name = upgrade_help.get('families', {}).get('rename', {})[name]
+            families[normalize_family(name)].attrib['help'] = text
+        for auto in autos:
+            if auto in variables:
+                variables[auto].attrib['hidden'] = 'True'
+            else:
+                # redefine value in first family
+                family = next(iter(families.values()))
+                variable = SubElement(family, 'variable', name=auto, redefine="True", hidden="True")
+                if self.test:
+                    del variable.attrib['redefine']
+        return root
+
+    @staticmethod
+    def move(elt, src, dst, optional=False):
+        if src == 'text':
+            value = elt.text
+            elt.text = None
+        else:
+            if optional and src not in elt.attrib:
+                return
+            value = elt.attrib[src]
+            del elt.attrib[src]
+        #
+        if dst == 'text':
+            elt.text = value
+        else:
+            elt.attrib[dst] = value
+
+    @staticmethod
+    def remove(elt, src, optional=False):
+        if optional and src not in elt.attrib:
+            return
+        del elt.attrib[src]
+
+    @staticmethod
+    def create_service(services, service_name, service_elt, servicelists, upgrade_help):
+        if service_name in service_elt:
+            return service_elt[service_name]
+        service = SubElement(services, 'service')
+        service.attrib['name'] = service_name
+        if service_name == 'unknown':
+            service.attrib['manage'] = 'False'
+        if service_name in upgrade_help.get('services', {}).get('unmanage', []):
+            service.attrib['manage'] = 'False'
+        service_elt[service_name] = service
+        if upgrade_help.get('servicelists', {}).get(service_name):
+            service.attrib['servicelist'] = upgrade_help.get('servicelists', {}).get(service_name)
+        elif service_name in servicelists:
+            service.attrib['servicelist'] = servicelists[service_name]
+        return service
+
+    def upgrade_container(self, elt, current_service, files, ips, servicelists, upgrade_help):
+        if elt.tag == 'file':
+            self.move(elt, 'name', 'text')
+            self.remove(elt, 'del_comment', optional=True)
+            elt.attrib['engine'] = 'creole_legacy'
+            if (not 'instance_mode' in elt.attrib or elt.attrib['instance_mode'] != 'when_container') and \
+                    elt.text not in upgrade_help.get('files', {}).get('remove', {}):
+                if elt.attrib.get('filelist') in upgrade_help.get('services', {}).get('filelist_service', {}):
+                    elt_service = upgrade_help.get('services', {}).get('filelist_service', {})[elt.attrib['filelist']]
+                    if elt_service in files:
+                        service = elt_service
+                    else:
+                        service = current_service
+                else:
+                    service = current_service
+                files[service][elt.text] = elt
+        elif elt.tag in ['host', 'disknod', 'fstab', 'interface', 'package', 'service_access']:
+            pass
+        elif elt.tag == 'service_restriction':
+            for restriction in elt:
+                if restriction.tag == 'ip' and restriction.text != '0.0.0.0':
+                    self.remove(restriction, 'ip_type', optional=True)
+                    self.remove(restriction, 'netmask_type', optional=True)
+                    if elt.attrib['service'] in upgrade_help.get('services', {}).get('rename', {}):
+                        elt_service = upgrade_help.get('services', {}).get('rename', {})[elt.attrib['service']]
+                    else:
+                        elt_service = elt.attrib['service']
+                    if elt_service in ips:
+                        service = elt_service
+                    else:
+                        service = current_service
+                    ips[service].append(restriction)
+        elif elt.tag == 'service':
+            new_name = elt.text
+            if current_service == 'unknown':
+                if new_name in files:
+                    raise Exception('hu?')
+                files[new_name] = files[current_service]
+                del files[current_service]
+                ips[new_name] = ips[current_service]
+                del ips[current_service]
+            elif new_name not in files:
+                files[new_name] = {}
+                ips[new_name] = []
+            current_service = new_name
+            if 'servicelist' in elt.attrib:
+                servicelists[current_service] = elt.attrib['servicelist']
+        else:
+            raise Exception(f'unknown containers tag {elt.tag}')
+        return current_service
+
+
+def is_empty(elt, not_check_attrib=False):
+    if not not_check_attrib and elt.attrib:
+        return False
+    empty = True
+    for file_ in elt:
+        empty = False
+    return empty
+
+
+def update_creole_1():
+    print('pfff')

src/rougail/utils.py

@@ -1,23 +1,66 @@
+"""Rougail's tools
+
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 """
-utilitaires créole
+from typing import List
-"""
+from unicodedata import normalize, combining
-import unicodedata
+import re
+
+from importlib.machinery import SourceFileLoader
+from importlib.util import spec_from_loader, module_from_spec
+
 from .i18n import _
+from .error import DictConsistencyError
+
+NAME_REGEXP = re.compile(r"^[a-z0-9_]*$")
 
 
-def normalize_family(family_name: str,
+def valid_variable_family_name(name: str,
-                     check_name: bool=True,
+                               xmlfiles: List[str],
-                     allow_dot: bool=False) -> str:
+                               ) -> None:
+    match = NAME_REGEXP.search(name)
+    if not match:
+        msg = _(f'invalid variable or family name "{name}" must only contains '
+                'lowercase ascii character, number or _')
+        raise DictConsistencyError(msg, 76, xmlfiles)
+
+
+def normalize_family(family_name: str) -> str:
     """replace space, accent, uppercase, ... by valid character
     """
-    family_name = family_name.replace('-', '_')
+    family_name = family_name.replace('-', '_').replace(' ', '_').replace('.', '_')
-    if not allow_dot:
+    nfkd_form = normalize('NFKD', family_name)
-        family_name = family_name.replace('.', '_')
+    family_name = ''.join([c for c in nfkd_form if not combining(c)])
-    family_name = family_name.replace(' ', '_')
+    return family_name.lower()
-    nfkd_form = unicodedata.normalize('NFKD', family_name)
-    family_name = ''.join([c for c in nfkd_form if not unicodedata.combining(c)])
-    family_name = family_name.lower()
-    if check_name and family_name == 'containers':
-        raise ValueError(_(f'"{family_name}" is a forbidden family name'))
-    return family_name
 
+
+def load_modules(eosfunc_file) -> List[str]:
+    """list all functions in eosfunc
+    """
+    loader = SourceFileLoader('eosfunc', eosfunc_file)
+    spec = spec_from_loader(loader.name, loader)
+    eosfunc = module_from_spec(spec)
+    loader.exec_module(eosfunc)
+    return eosfunc

src/rougail/xml_compare.py

@@ -1,161 +0,0 @@
-try:
-    import doctest
-    doctest.OutputChecker
-except (AttributeError, ImportError): # Python < 2.4
-    import util.doctest24 as doctest
-try:
-    import xml.etree.ElementTree as ET
-except ImportError:
-    import elementtree.ElementTree as ET
-from xml.parsers.expat import ExpatError as XMLParseError
-
-RealOutputChecker = doctest.OutputChecker
-
-
-def debug(*msg):
-    import sys
-    print >> sys.stderr, ' '.join(map(str, msg))
-
-
-class HTMLOutputChecker(RealOutputChecker):
-
-    def check_output(self, want, got, optionflags):
-        normal = RealOutputChecker.check_output(self, want, got, optionflags)
-        if normal or not got:
-            return normal
-        try:
-            want_xml = make_xml(want)
-        except XMLParseError:
-            pass
-        else:
-            try:
-                got_xml = make_xml(got)
-            except XMLParseError:
-                pass
-            else:
-                if xml_compare(want_xml, got_xml):
-                    return True
-        return False
-
-    def output_difference(self, example, got, optionflags):
-        actual = RealOutputChecker.output_difference(
-            self, example, got, optionflags)
-        want_xml = got_xml = None
-        try:
-            want_xml = make_xml(example.want)
-            want_norm = make_string(want_xml)
-        except XMLParseError as e:
-            if example.want.startswith('<'):
-                want_norm = '(bad XML: %s)' % e
-                #  '<xml>%s</xml>' % example.want
-            else:
-                return actual
-        try:
-            got_xml = make_xml(got)
-            got_norm = make_string(got_xml)
-        except XMLParseError as e:
-            if example.want.startswith('<'):
-                got_norm = '(bad XML: %s)' % e
-            else:
-                return actual
-        s = '%s\nXML Wanted: %s\nXML Got   : %s\n' % (
-            actual, want_norm, got_norm)
-        if got_xml and want_xml:
-            result = []
-            xml_compare(want_xml, got_xml, result.append)
-            s += 'Difference report:\n%s\n' % '\n'.join(result)
-        return s
-
-
-def xml_sort(children):
-    tcl1 = {}
-    #idx = 0
-
-    for child in children:
-        if 'name' in child.attrib:
-            key = child.attrib['name']
-        else:
-            key = child.tag
-        if key not in tcl1:
-            tcl1[key] = []
-        tcl1[key].append(child)
-    cl1_keys = list(tcl1.keys())
-    cl1_keys.sort()
-    cl1 = []
-    for key in cl1_keys:
-        cl1.extend(tcl1[key])
-    return cl1
-
-def xml_compare(x1, x2):
-    if x1.tag != x2.tag:
-        print ('Tags do not match: %s and %s' % (x1.tag, x2.tag))
-        return False
-    for name, value in x1.attrib.items():
-        if x2.attrib.get(name) != value:
-            print ('Attributes do not match: %s=%r, %s=%r'
-                     % (name, value, name, x2.attrib.get(name)))
-            return False
-    for name in x2.attrib:
-        if name not in x1.attrib:
-            print ('x2 has an attribute x1 is missing: %s'
-                     % name)
-            return False
-    if not text_compare(x1.text, x2.text):
-        print ('text: %r != %r' % (x1.text, x2.text))
-        return False
-    if not text_compare(x1.tail, x2.tail):
-        print ('tail: %r != %r' % (x1.tail, x2.tail))
-        return False
-
-    cl1 = xml_sort(x1.getchildren())
-    cl2 = xml_sort(x2.getchildren())
-
-    if len(cl1) != len(cl2):
-        cl1_tags = []
-        for c in cl1:
-            cl1_tags.append(c.tag)
-        cl2_tags = []
-        for c in cl2:
-            cl2_tags.append(c.tag)
-        print ('children length differs, %i != %i (%s != %s)'
-                 % (len(cl1), len(cl2), cl1_tags, cl2_tags))
-        return False
-    i = 0
-    for c1, c2 in zip(cl1, cl2):
-        i += 1
-        if not xml_compare(c1, c2):
-            if 'name' in c1.attrib:
-                name = c1.attrib['name']
-            else:
-                name = i
-            print ('in tag "%s" with name "%s"'
-                     % (c1.tag, name))
-            return False
-    return True
-
-
-def text_compare(t1, t2):
-    if not t1 and not t2:
-        return True
-    if t1 == '*' or t2 == '*':
-        return True
-    return (t1 or '').strip() == (t2 or '').strip()
-
-
-def make_xml(s):
-    return ET.XML('<xml>%s</xml>' % s)
-
-
-def make_string(xml):
-    if isinstance(xml, (str, unicode)):
-        xml = make_xml(xml)
-    s = ET.tostring(xml)
-    if s == '<xml />':
-        return ''
-    assert s.startswith('<xml>') and s.endswith('</xml>'), repr(s)
-    return s[5:-6]
-
-
-def install():
-    doctest.OutputChecker = HTMLOutputChecker
-

src/rougail/xmlreflector.py

@@ -1,90 +1,77 @@
-# coding: utf-8
+"""load XML file from directory
-from os.path import join, isfile, basename, isdir
-from os import listdir
-#from io import BytesIO
 
-from lxml.etree import DTD, parse, tostring  # , XMLParser
+Created by:
+EOLE (http://eole.orion.education.fr)
+Copyright (C) 2005-2018
+
+Forked by:
+Cadoles (http://www.cadoles.com)
+Copyright (C) 2019-2021
+
+distribued with GPL-2 or later license
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+"""
+from typing import List
+from os.path import join, isfile
+from os import listdir
+
+from lxml.etree import DTD, parse, XMLSyntaxError  # pylint: disable=E0611
 
 from .i18n import _
 from .error import DictConsistencyError
 
 
-class XMLReflector(object):
+class XMLReflector:
     """Helper class for loading the Creole XML file,
     parsing it, validating against the Creole DTD,
     writing the xml result on the disk
     """
-    def __init__(self):
+    def __init__(self,
-        self.dtd = None
+                 rougailconfig: 'RougailConfig',
-
+                 ) -> None:
-    def parse_dtd(self, dtdfilename):
         """Loads the Creole DTD
 
         :raises IOError: if the DTD is not found
 
         :param dtdfilename: the full filename of the Creole DTD
         """
+        dtdfilename = rougailconfig['dtdfilename']
         if not isfile(dtdfilename):
             raise IOError(_("no such DTD file: {}").format(dtdfilename))
         with open(dtdfilename, 'r') as dtdfd:
             self.dtd = DTD(dtdfd)
 
-    def parse_xmlfile(self, xmlfile):
+    def load_xml_from_folders(self,
-        """Parses and validates some Creole XML against the Creole DTD
+                              xmlfolders: List[str],
-
+                              ):
-        :returns: the root element tree object
-        """
-        # document = parse(BytesIO(xmlfile), XMLParser(remove_blank_text=True))
-        document = parse(xmlfile)
-        if not self.dtd.validate(document):
-            raise DictConsistencyError(_(f'"{xmlfile}" not a valid xml file: {self.dtd.error_log.filter_from_errors()[0]}'))
-        return document.getroot()
-
-    def load_xml_from_folders(self, xmlfolders):
         """Loads all the XML files located in the xmlfolders' list
 
         :param xmlfolders: list of full folder's name
         """
-        documents = []
-        if not isinstance(xmlfolders, list):
-            xmlfolders = [xmlfolders]
         for xmlfolder in xmlfolders:
-            if isinstance(xmlfolder, list) or isinstance(xmlfolder, tuple):
+            filenames = [join(xmlfolder, filename) for filename in listdir(xmlfolder) if \
-                # directory group : collect files from each
+                         filename.endswith('.xml')]
-                # directory and sort them before loading
-                group_files = []
-                for idx, subdir in enumerate(xmlfolder):
-                    if isdir(subdir):
-                        for filename in listdir(subdir):
-                            group_files.append((filename, idx, subdir))
-                    else:
-                        group_files.append(basename(subdir), idx, dirname(subdir))
-                def sort_group(file1, file2):
-                    if file1[0] == file2[0]:
-                        # sort by initial xmlfolder order if same name
-                        return file1[1].__cmp__(file2[1])
-                    # sort by filename
-                    elif file1[0] > file2[0]:
-                        return 1
-                    else:
-                        return -1
-                group_files.sort(sort_group)
-                filenames = [join(f[2], f[0]) for f in group_files]
-            elif isdir(xmlfolder):
-                filenames = []
-                for filename in listdir(xmlfolder):
-                    filenames.append(join(xmlfolder, filename))
             filenames.sort()
-            else:
-                filenames = [xmlfolder]
             for xmlfile in filenames:
-                if xmlfile.endswith('.xml'):
+                try:
-                    #xmlfile_path = join(xmlfolder, xmlfile)
+                    document = parse(xmlfile)
-                    documents.append((xmlfile, self.parse_xmlfile(xmlfile)))
+                except XMLSyntaxError as err:
-        return documents
+                    raise DictConsistencyError(_(f'not a XML file: {err}'), 52, [xmlfile]) from err
-
+                if not self.dtd.validate(document):
-    def save_xmlfile(self, xmlfilename, xml):  # pylint: disable=R0201
+                    dtd_error = self.dtd.error_log.filter_from_errors()[0]
-        """Write a bunch of XML on the disk
+                    msg = _(f'not a valid XML file: {dtd_error}')
-        """
+                    raise DictConsistencyError(msg, 43, [xmlfile])
-        with open(xmlfilename, 'w') as xmlfh:
+                yield xmlfile, document.getroot()
-            xmlfh.write(tostring(xml, pretty_print=True, encoding="UTF-8", xml_declaration=True).decode('utf8'))

tests/dictionaries/00empty/00_base.xml

@@ -1,5 +1,5 @@
 <?xml version='1.0' encoding='UTF-8'?>
-<rougail>
+<rougail version="0.9">
     <services>
         <service name="tata">
         </service>

tests/dictionaries/00empty/makedict/after.json

@@ -0,0 +1,10 @@
+{
+    "services.tata.activate": {
+        "owner": "default",
+        "value": true
+    },
+    "services.tata.manage": {
+        "owner": "default",
+        "value": true
+    }
+}

tests/dictionaries/00empty/makedict/base.json

@@ -0,0 +1,4 @@
+{
+    "services.tata.activate": true,
+    "services.tata.manage": true
+}

tests/dictionaries/00empty/makedict/before.json

@@ -0,0 +1,10 @@
+{
+    "services.tata.activate": {
+        "owner": "default",
+        "value": true
+    },
+    "services.tata.manage": {
+        "owner": "default",
+        "value": true
+    }
+}

tests/dictionaries/00empty/tiramisu/base.py

@@ -1,14 +1,18 @@
-import imp
+from importlib.machinery import SourceFileLoader
-func = imp.load_source('func', 'tests/dictionaries/../eosfunc/test.py')
+from importlib.util import spec_from_loader, module_from_spec
+loader = SourceFileLoader('func', 'tests/dictionaries/../eosfunc/test.py')
+spec = spec_from_loader(loader.name, loader)
+func = module_from_spec(spec)
+loader.exec_module(func)
 for key, value in dict(locals()).items():
-    if key != ['imp', 'func']:
+    if key != ['SourceFileLoader', 'func']:
         setattr(func, key, value)
 try:
     from tiramisu3 import *
 except:
     from tiramisu import *
-from rougail.tiramisu import ConvertDynOptionDescription
+option_3 = BoolOption(name="activate", doc="activate", default=True)
-option_2 = OptionDescription(name='tata', doc='tata', children=[])
+option_4 = BoolOption(name="manage", doc="manage", default=True)
-option_2.impl_set_information("manage", True)
+option_2 = OptionDescription(name="tata", doc="tata", children=[option_3, option_4])
-option_1 = OptionDescription(name='services', doc='services', properties=frozenset({'hidden'}), children=[option_2])
+option_1 = OptionDescription(name="services", doc="services", children=[option_2], properties=frozenset({"hidden"}))
-option_0 = OptionDescription(name='baseoption', doc='baseoption', children=[option_1])
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[option_1])

tests/dictionaries/00load_autofreeze/00-base.xml

@@ -1,23 +1,13 @@
 <?xml version='1.0' encoding='UTF-8'?>
-<rougail>
+<rougail version="0.9">
-
     <variables>
-        <family name="général">
+        <variable name="myvar" auto_freeze="True">
-            <variable name="mode_conteneur_actif" type="oui/non" description="No change" auto_freeze="True">
+            <value>no</value>
-                <value>non</value>
         </variable>
-            <variable name="module_instancie" type="oui/non" description="No change">
+        <variable name="server_deployed" type="boolean">
-                <value>non</value>
+            <value>False</value>
         </variable>
-        </family>
-        <separators/>
     </variables>
-
-    <constraints>
-    </constraints>
-
-    <help/>
-
 </rougail>
 <!-- vim: ts=4 sw=4 expandtab
 -->

tests/dictionaries/00load_autofreeze/makedict/after.json

@@ -0,0 +1,10 @@
+{
+    "rougail.myvar": {
+        "owner": "forced",
+        "value": "no"
+    },
+    "rougail.server_deployed": {
+        "owner": "default",
+        "value": false
+    }
+}

tests/dictionaries/00load_autofreeze/makedict/base.json

@@ -1 +1,4 @@
-{"rougail.general.mode_conteneur_actif": "non", "rougail.general.module_instancie": "non"}
+{
+    "rougail.myvar": "no",
+    "rougail.server_deployed": false
+}

tests/dictionaries/00load_autofreeze/makedict/before.json

@@ -0,0 +1,10 @@
+{
+    "rougail.myvar": {
+        "owner": "default",
+        "value": "no"
+    },
+    "rougail.server_deployed": {
+        "owner": "default",
+        "value": false
+    }
+}

tests/dictionaries/00load_autofreeze/tiramisu/base.py

@@ -1,15 +1,17 @@
-import imp
+from importlib.machinery import SourceFileLoader
-func = imp.load_source('func', 'tests/dictionaries/../eosfunc/test.py')
+from importlib.util import spec_from_loader, module_from_spec
+loader = SourceFileLoader('func', 'tests/dictionaries/../eosfunc/test.py')
+spec = spec_from_loader(loader.name, loader)
+func = module_from_spec(spec)
+loader.exec_module(func)
 for key, value in dict(locals()).items():
-    if key != ['imp', 'func']:
+    if key != ['SourceFileLoader', 'func']:
         setattr(func, key, value)
 try:
     from tiramisu3 import *
 except:
     from tiramisu import *
-from rougail.tiramisu import ConvertDynOptionDescription
+option_3 = BoolOption(name="server_deployed", doc="server_deployed", default=False, properties=frozenset({"mandatory", "normal"}))
-option_4 = ChoiceOption(properties=frozenset({'mandatory', 'normal'}), name='module_instancie', doc='No change', multi=False, default='non', values=('oui', 'non'))
+option_2 = StrOption(name="myvar", doc="myvar", default="no", properties=frozenset({"basic", "force_store_value", "mandatory", Calculation(func.calc_value, Params(ParamValue('frozen'), kwargs={'condition': ParamOption(option_3, todict=True, notraisepropertyerror=True), 'expected': ParamValue(True)}))}))
-option_3 = ChoiceOption(properties=frozenset({'auto_freeze', 'basic', 'force_store_value', 'mandatory', Calculation(calc_value, Params(ParamValue('auto_frozen'), kwargs={'condition': ParamOption(option_4, todict=True), 'expected': ParamValue('oui'), 'reverse_condition': ParamValue(True)}))}), name='mode_conteneur_actif', doc='No change', multi=False, default='non', values=('oui', 'non'))
+option_1 = OptionDescription(name="rougail", doc="rougail", children=[option_2, option_3])
-option_2 = OptionDescription(name='general', doc='général', properties=frozenset({'basic'}), children=[option_3, option_4])
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[option_1])
-option_1 = OptionDescription(name='rougail', doc='rougail', children=[option_2])
-option_0 = OptionDescription(name='baseoption', doc='baseoption', children=[option_1])

tests/dictionaries/00load_autofreezeexpert/00-base.xml

@@ -1,25 +1,13 @@
 <?xml version='1.0' encoding='UTF-8'?>
-<rougail>
+<rougail version="0.9">
-
-    <services/>
-
     <variables>
-        <family name="général">
+        <variable name="my_var" auto_freeze="True" mode="expert">
-            <variable name="mode_conteneur_actif" type="oui/non" description="No change" auto_freeze="True" mode="expert">
+            <value>no</value>
-                <value>non</value>
         </variable>
-            <variable name="module_instancie" type="oui/non" description="No change">
+        <variable name="server_deployed" type="boolean">
-                <value>non</value>
+            <value>False</value>
         </variable>
-        </family>
-        <separators/>
     </variables>
-
-    <constraints>
-    </constraints>
-
-    <help/>
-
 </rougail>
 <!-- vim: ts=4 sw=4 expandtab
 -->

tests/dictionaries/00load_autofreezeexpert/makedict/after.json

@@ -0,0 +1,10 @@
+{
+    "rougail.my_var": {
+        "owner": "forced",
+        "value": "no"
+    },
+    "rougail.server_deployed": {
+        "owner": "default",
+        "value": false
+    }
+}

tests/dictionaries/00load_autofreezeexpert/makedict/base.json

@@ -1 +1,4 @@
-{"rougail.general.mode_conteneur_actif": "non", "rougail.general.module_instancie": "non"}
+{
+    "rougail.my_var": "no",
+    "rougail.server_deployed": false
+}

tests/dictionaries/00load_autofreezeexpert/makedict/before.json

@@ -0,0 +1,10 @@
+{
+    "rougail.my_var": {
+        "owner": "default",
+        "value": "no"
+    },
+    "rougail.server_deployed": {
+        "owner": "default",
+        "value": false
+    }
+}

tests/dictionaries/00load_autofreezeexpert/tiramisu/base.py

@@ -1,15 +1,17 @@
-import imp
+from importlib.machinery import SourceFileLoader
-func = imp.load_source('func', 'tests/dictionaries/../eosfunc/test.py')
+from importlib.util import spec_from_loader, module_from_spec
+loader = SourceFileLoader('func', 'tests/dictionaries/../eosfunc/test.py')
+spec = spec_from_loader(loader.name, loader)
+func = module_from_spec(spec)
+loader.exec_module(func)
 for key, value in dict(locals()).items():
-    if key != ['imp', 'func']:
+    if key != ['SourceFileLoader', 'func']:
         setattr(func, key, value)
 try:
     from tiramisu3 import *
 except:
     from tiramisu import *
-from rougail.tiramisu import ConvertDynOptionDescription
+option_3 = BoolOption(name="server_deployed", doc="server_deployed", default=False, properties=frozenset({"mandatory", "normal"}))
-option_4 = ChoiceOption(properties=frozenset({'mandatory', 'normal'}), name='module_instancie', doc='No change', multi=False, default='non', values=('oui', 'non'))
+option_2 = StrOption(name="my_var", doc="my_var", default="no", properties=frozenset({"expert", "force_store_value", "mandatory", Calculation(func.calc_value, Params(ParamValue('frozen'), kwargs={'condition': ParamOption(option_3, todict=True, notraisepropertyerror=True), 'expected': ParamValue(True)}))}))
-option_3 = ChoiceOption(properties=frozenset({'auto_freeze', 'expert', 'force_store_value', 'mandatory', Calculation(calc_value, Params(ParamValue('auto_frozen'), kwargs={'condition': ParamOption(option_4, todict=True), 'expected': ParamValue('oui'), 'reverse_condition': ParamValue(True)}))}), name='mode_conteneur_actif', doc='No change', multi=False, default='non', values=('oui', 'non'))
+option_1 = OptionDescription(name="rougail", doc="rougail", children=[option_2, option_3])
-option_2 = OptionDescription(name='general', doc='général', properties=frozenset({'normal'}), children=[option_3, option_4])
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[option_1])
-option_1 = OptionDescription(name='rougail', doc='rougail', children=[option_2])
-option_0 = OptionDescription(name='baseoption', doc='baseoption', children=[option_1])

tests/dictionaries/00load_autosave/00-base.xml

@@ -1,22 +1,13 @@
 <?xml version='1.0' encoding='UTF-8'?>
-<rougail>
+<rougail version="0.9">
-
-    <services/>
-
     <variables>
-        <family name="général">
+        <variable name="server_deployed" type="boolean"/>
-            <variable name="mode_conteneur_actif" type="oui/non" description="No change" auto_save="True">
+        <family name="general" description="général">
+            <variable name="mode_conteneur_actif" type="string" description="No change" auto_save="True">
                 <value>non</value>
             </variable>
         </family>
-        <separators/>
     </variables>
-
-    <constraints>
-    </constraints>
-
-    <help/>
-
 </rougail>
 <!-- vim: ts=4 sw=4 expandtab
 -->

tests/dictionaries/00load_autosave/makedict/after.json

@@ -0,0 +1,10 @@
+{
+    "rougail.server_deployed": {
+        "owner": "default",
+        "value": true
+    },
+    "rougail.general.mode_conteneur_actif": {
+        "owner": "forced",
+        "value": "non"
+    }
+}

tests/dictionaries/00load_autosave/makedict/base.json

@@ -1 +1,4 @@
-{"rougail.general.mode_conteneur_actif": "non"}
+{
+    "rougail.server_deployed": true,
+    "rougail.general.mode_conteneur_actif": "non"
+}

tests/dictionaries/00load_autosave/makedict/before.json

@@ -0,0 +1,10 @@
+{
+    "rougail.server_deployed": {
+        "owner": "default",
+        "value": true
+    },
+    "rougail.general.mode_conteneur_actif": {
+        "owner": "default",
+        "value": "non"
+    }
+}

tests/dictionaries/00load_autosave/tiramisu/base.py

@@ -1,14 +1,18 @@
-import imp
+from importlib.machinery import SourceFileLoader
-func = imp.load_source('func', 'tests/dictionaries/../eosfunc/test.py')
+from importlib.util import spec_from_loader, module_from_spec
+loader = SourceFileLoader('func', 'tests/dictionaries/../eosfunc/test.py')
+spec = spec_from_loader(loader.name, loader)
+func = module_from_spec(spec)
+loader.exec_module(func)
 for key, value in dict(locals()).items():
-    if key != ['imp', 'func']:
+    if key != ['SourceFileLoader', 'func']:
         setattr(func, key, value)
 try:
     from tiramisu3 import *
 except:
     from tiramisu import *
-from rougail.tiramisu import ConvertDynOptionDescription
+option_2 = BoolOption(name="server_deployed", doc="server_deployed", default=True, properties=frozenset({"mandatory", "normal"}))
-option_3 = ChoiceOption(properties=frozenset({'basic', 'force_store_value', 'mandatory'}), name='mode_conteneur_actif', doc='No change', multi=False, default='non', values=('oui', 'non'))
+option_4 = StrOption(name="mode_conteneur_actif", doc="No change", default="non", properties=frozenset({"basic", "force_store_value", "mandatory"}))
-option_2 = OptionDescription(name='general', doc='général', properties=frozenset({'basic'}), children=[option_3])
+option_3 = OptionDescription(name="general", doc="général", children=[option_4], properties=frozenset({"basic"}))
-option_1 = OptionDescription(name='rougail', doc='rougail', children=[option_2])
+option_1 = OptionDescription(name="rougail", doc="rougail", children=[option_2, option_3])
-option_0 = OptionDescription(name='baseoption', doc='baseoption', children=[option_1])
+option_0 = OptionDescription(name="baseoption", doc="baseoption", children=[option_1])

tests/dictionaries/00load_autosaveexpert/00-base.xml

@@ -1,22 +1,13 @@
 <?xml version='1.0' encoding='UTF-8'?>
-<rougail>
+<rougail version="0.9">
-
-    <services/>
-
     <variables>
-        <family name="général">
+        <variable name="server_deployed" type="boolean"/>
-            <variable name="mode_conteneur_actif" type="oui/non" description="No change" auto_save="True" mode="expert">
+        <family name="general" description="général">
+            <variable name="mode_conteneur_actif" type="string" description="No change" auto_save="True" mode="expert">
                 <value>non</value>
             </variable>
         </family>
-        <separators/>
     </variables>
-
-    <constraints>
-    </constraints>
-
-    <help/>
-
 </rougail>
 <!-- vim: ts=4 sw=4 expandtab
 -->

tests/dictionaries/00load_autosaveexpert/makedict/after.json

@@ -0,0 +1,10 @@
+{
+    "rougail.server_deployed": {
+        "owner": "default",
+        "value": true
+    },
+    "rougail.general.mode_conteneur_actif": {
+        "owner": "forced",
+        "value": "non"
+    }
+}