Skip to content

Configuration

Soda configuration block

SodaConfiguration

Bases: Block

This block can be used to provide the configuration required to run Soda scans. For more information, please refer to the official docs # noqa

Parameters:

Name Type Description Default
configuration_yaml_path str

Absolute path of the Soda configuration file.

 required
configuration_yaml_str str

Optional YAML string containing the Soda configuration details. If provided, it will be saved at the path provided with configuration_yaml_path.

 required
Example

Load stored Soda configuration. 

from prefect_soda_core.soda_configuration import SodaConfiguration
soda_configuration_block = SodaConfiguration.load("BLOCK_NAME")

Source code in prefect_soda_core/soda_configuration.py 
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
class SodaConfiguration(Block):
    """
    This block can be used to provide the configuration
    required to run Soda scans.
    For more information, please refer to the
    [official docs](https://docs.soda.io/soda-core/configuration.html#configuration-instructions)  # noqa

    Args:
        configuration_yaml_path (str): Absolute path of the Soda configuration file.
        configuration_yaml_str (str): Optional YAML string containing the Soda configuration
            details. If provided, it will be saved
            at the path provided with `configuration_yaml_path`.

    Example:
        Load stored Soda configuration.
        ```python
        from prefect_soda_core.soda_configuration import SodaConfiguration
        soda_configuration_block = SodaConfiguration.load("BLOCK_NAME")
        ```
    """

    configuration_yaml_path: str
    configuration_yaml_str: Optional[str]

    _block_type_name: Optional[str] = "Soda Configuration"
    _logo_url: Optional[
        HttpUrl
    ] = "https://raw.githubusercontent.com/PrefectHQ/prefect/main/docs/img/collections/soda.png"  # noqa

    @root_validator(pre=True)
    def check_block_configuration(cls, values):
        """
        Ensure that the configuration options are valid.
        A configuration is valid if it provides just the path to the
        YAML configuration file or if it has both the path
        to the configuration file and a valid YAML configuration string.

        Raises:
            SodaConfigurationException: When the provided configuration is not valid.
        """
        configuration_yaml_str_exists = bool(values.get("configuration_yaml_str"))

        # If the YAML string is passed, but is not a valid YAML, then raise error
        if configuration_yaml_str_exists:
            try:
                yaml_str = values.get("configuration_yaml_str")
                safe_load(yaml_str)
            except YAMLError as exc:
                msg = f"The provided configuration YAML is not valid. Error is: {exc}"
                raise SodaConfigurationException(msg)

        return values

    def persist_configuration(self):
        """
        Persist Soda configuration on the file system, if necessary.
        Please note that, if the path already exists, it will be overwritten.
        """

        # If a YAML string and path are passed, then persist the configuration
        if self.configuration_yaml_str and self.configuration_yaml_path:
            with open(self.configuration_yaml_path, "w") as f:
                safe_dump(data=self.configuration_yaml_str, stream=f)

check_block_configuration(values)

Ensure that the configuration options are valid. A configuration is valid if it provides just the path to the YAML configuration file or if it has both the path to the configuration file and a valid YAML configuration string.

Raises:

Type Description
SodaConfigurationException

When the provided configuration is not valid.
Source code in prefect_soda_core/soda_configuration.py 
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
@root_validator(pre=True)
def check_block_configuration(cls, values):
    """
    Ensure that the configuration options are valid.
    A configuration is valid if it provides just the path to the
    YAML configuration file or if it has both the path
    to the configuration file and a valid YAML configuration string.

    Raises:
        SodaConfigurationException: When the provided configuration is not valid.
    """
    configuration_yaml_str_exists = bool(values.get("configuration_yaml_str"))

    # If the YAML string is passed, but is not a valid YAML, then raise error
    if configuration_yaml_str_exists:
        try:
            yaml_str = values.get("configuration_yaml_str")
            safe_load(yaml_str)
        except YAMLError as exc:
            msg = f"The provided configuration YAML is not valid. Error is: {exc}"
            raise SodaConfigurationException(msg)

    return values

persist_configuration()

Persist Soda configuration on the file system, if necessary. Please note that, if the path already exists, it will be overwritten.

Source code in prefect_soda_core/soda_configuration.py 
65
66
67
68
69
70
71
72
73
74
def persist_configuration(self):
    """
    Persist Soda configuration on the file system, if necessary.
    Please note that, if the path already exists, it will be overwritten.
    """

    # If a YAML string and path are passed, then persist the configuration
    if self.configuration_yaml_str and self.configuration_yaml_path:
        with open(self.configuration_yaml_path, "w") as f:
            safe_dump(data=self.configuration_yaml_str, stream=f)