Example Validation
Polars
gQzUyLjY1MjM0NCw1NCA1NCw1Mi42NTIzNDQgNTQsNTEgTDU0LDEzIEM1NCwxMS4zNDc2NTYgNTIuNjUyMzQ0LDEwIDUxLDEwIFogTTM4LjcwNzAzMSw0NS4yOTI5NjkgTDM3LjI5Mjk2OSw0Ni43MDcwMzEgTDIyLjU4NTkzOCwzMiBMMzcuMjkyOTY5LDE3LjI5Mjk2OSBMMzguNzA3MDMxLDE4LjcwNzAzMSBMMjUuNDE0MDYzLDMyIEwzOC43MDcwMzEsNDUuMjkyOTY5IFoiIGlkPSJsZXNzX3RoYW4iIGZpbGw9IiMwMDAwMDAiIC8+CiAgICAgICAgPC9nPgogICAgPC9nPgo8L3N2Zz4=" />
col_vals_lt()
1.00
0.00
col_vals_between()
0.92
0.08
col_vals_in_set()
1.00
0.00
col_vals_regex()
1.00
0.00
There are three things that should be noted here:
- validation steps: each step is a separate test on the table, focused on a certain aspect of the table
- validation rules: the validation type is provided here along with key constraints
- validation results: interrogation results are provided here, with a breakdown of test units (*total*, *passing*, and *failing*), threshold flags, and more
The intent is to provide the key information in one place, and have it be interpretable by data stakeholders. For example, a failure can be seen in the second row (notice there's a CSV button). A data quality stakeholder could click this to download a CSV of the failing rows for that step.
# Example Code, Step-by-Step
This section will walk you through the example code used above.
``` python
import pointblank as pb
(
pb.Validate(data=pb.load_dataset(dataset="small_table"))
.col_vals_lt(columns="a", value=10)
.col_vals_between(columns="d", left=0, right=5000)
.col_vals_in_set(columns="f", set=["low", "mid", "high"])
.col_vals_regex(columns="b", pattern=r"^[0-9]-[a-z]{3}-[0-9]{3}$")
.interrogate()
)
```
Note these three key pieces in the code:
- **data**: the `Validate(data=)` argument takes a DataFrame or database table that you want to validate
- **steps**: the methods starting with `col_vals_` specify validation steps that run on specific columns
- **execution**: the 
Test units are dependent on the test being run. Some validation methods might test every value in a particular column, so each value will be a test unit. Others will only have a single test unit since they aren't testing individual values but rather if the overall test passes or fails.
# Setting Thresholds for Data Quality Signals
Understanding test units is essential because they form the foundation of Pointblank's threshold system. Thresholds let you define acceptable levels of data quality, triggering different severity signals ('warning', 'error', or 'critical') when certain failure conditions are met.
Here's a simple example that uses a single validation step along with thresholds set using the [Thresholds](../../reference/Thresholds.md#pointblank.Thresholds) class:
``` python
(
pb.Validate(data=pb.load_dataset(dataset="small_table"))
.col_vals_lt(
columns="a",
value=7,
# Set the 'warning' and 'error' thresholds ---
thresholds=pb.Thresholds(warning=2, error=4)
)
.interrogate()
)
```
col_vals_lt()