col_sd_gt()
0.00
1.00
col() object referencing another column whose standard deviation will be used for comparison, (3) a ref() object referencing a column in reference data (when `Validate(reference=)` has been set), or (4) `None` to automatically compare against the same column in reference data (shorthand for `ref(column_name)` when reference data is set).
`tol: Tolerance = ``0`
A tolerance value for the comparison. The default is `0`, meaning exact comparison. When set to a positive value, the comparison becomes more lenient. For example, with `tol=0.5`, a standard deviation that differs from the target by up to `0.5` will still pass. The `tol=` parameter expands the acceptable range for the comparison. For [col_sd_gt()](Validate.col_sd_gt.md#pointblank.Validate.col_sd_gt), a tolerance of `tol=0.5` would mean the standard deviation can be within `0.5` of the target value and still pass validation.
`thresholds: float | bool | tuple | dict | Thresholds | None = None`
Failure threshold levels so that the validation step can react accordingly when failing test units are level. Since this is an aggregation-based validation with only one test unit, threshold values typically should be set as absolute counts (e.g., `1`) to indicate pass/fail, or as proportions where any value less than `1.0` means failure is acceptable.
`brief: str | bool = ``False`
An optional brief description of the validation step that will be displayed in the reporting table. You can use the templating elements like `"{step}"` to insert the step number, or `"{auto}"` to include an automatically generated brief. If `True` the entire brief will be automatically generated. If `None` (the default) then there won't be a brief.
`actions: Actions | None = None`
Optional actions to take when the validation step meets or exceeds any set threshold levels. If provided, the Actions class should be used to define the actions.
`active: bool | Callable = ``True`
A boolean value or callable that determines whether the validation step should be active. Using `False` will make the validation step inactive (still reporting its presence and keeping indexes for the steps unchanged). A callable can also be provided; it will receive the data table as its single argument and must return a boolean value. The callable is evaluated *before* any `pre=` processing. Inspection functions like has_columns() and has_rows() can be used here to conditionally activate a step based on properties of the target table.
## Returns
`Validate`
The [Validate](Validate.md#pointblank.Validate) object with the added validation step.
## Using Reference Data
The [col_sd_gt()](Validate.col_sd_gt.md#pointblank.Validate.col_sd_gt) method supports comparing column aggregations against reference data. This is useful for validating that statistical properties remain consistent across different versions of a dataset, or for comparing current data against historical baselines.
To use reference data, set the `reference=` parameter when creating the [Validate](Validate.md#pointblank.Validate) object:
``` python
validation = (
pb.Validate(data=current_data, reference=baseline_data)
.col_sd_gt(columns="revenue") # Compares sum(current.revenue) vs sum(baseline.revenue)
.interrogate()
)
```
When `value=None` and reference data is set, the method automatically compares against the same column in the reference data. You can also explicitly specify reference columns using the [ref()](ref.md#pointblank.ref) helper:
``` python
.col_sd_gt(columns="revenue", value=pb.ref("baseline_revenue"))
```
## Understanding Tolerance
The `tol=` parameter allows for fuzzy comparisons, which is especially important for floating-point aggregations where exact equality is often unreliable.
The `tol=` parameter expands the acceptable range for the comparison. For [col_sd_gt()](Validate.col_sd_gt.md#pointblank.Validate.col_sd_gt), a tolerance of `tol=0.5` would mean the standard deviation can be within `0.5` of the target value and still pass validation.
For equality comparisons (`col_*_eq`), the tolerance creates a range `[value - tol, value + tol]` within which the aggregation is considered valid. For inequality comparisons, the tolerance shifts the comparison boundary.
## Thresholds
The `thresholds=` parameter is used to set the failure-condition levels for the validation step. If they are set here at the step level, these thresholds will override any thresholds set at the global level in `Validate(thresholds=...)`.
There are three threshold levels: 'warning', 'error', and 'critical'. Since aggregation validations operate on a single test unit (the aggregated value), threshold values are typically set as absolute counts:
- `thresholds=1` means any failure triggers a 'warning'
- `thresholds=(1, 1, 1)` means any failure triggers all three levels
Thresholds can be defined using one of these input schemes:
1. use the Thresholds class (the most direct way to create thresholds)
2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is the 'error' level, and position `2` is the 'critical' level
3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and 'critical'
4. a single integer/float value denoting absolute number or fraction of failing test units for the 'warning' level only
## Examples
For the examples, we'll use a simple Polars DataFrame with numeric columns. The table is shown below:
``` python
import pointblank as pb
import polars as pl
tbl = pl.DataFrame(
{
"a": [1, 2, 3, 4, 5],
"b": [2, 2, 2, 2, 2],
}
)
pb.preview(tbl)
```
a
Int64
|
b
Int64
|
|
|---|---|---|
| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C66 | 1 |
col_sd_gt()
|
|
✓ | 1 | 0 0.00 |
1 1.00 |
-- | -- | -- | -- |
| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C66 | 1 |
col_sd_gt()
|
|
✓ | 1 | 0 0.00 |
1 1.00 |
-- | -- | -- | -- | ||
| #4CA64C66 | 2 |
col_sd_gt()
|
|
✓ | 1 | 0 0.00 |
1 1.00 |
-- | -- | -- | -- |
| STEP | COLUMNS | VALUES | TBL | EVAL | UNITS | PASS | FAIL | W | E | C | EXT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #4CA64C | 1 |
ODgsMTAuNzY2IFoiIGlkPSJzIiAvPgogICAgICAgICAgICAgICAgPHBhdGggZD0iTTExLjA4OCwxMC43NjYgQzEwLjMzMiwxMC43NjYgOS42NzQsMTAuNjIzNjY2NyA5LjExNCwxMC4zMzkgQzguNTU0LDEwLjA1NDMzMzMgOC4xMiw5LjY0NiA3LjgxMiw5LjExNCBDNy41MDQsOC41ODIgNy4zNSw3Ljk1MiA3LjM1LDcuMjI0IEM3LjM1LDYuNTI0IDcuNDk3LDUuOTAxIDcuNzkxLDUuMzU1IEM4LjA4NSw0LjgwOSA4LjUxNjY2NjY3LDQuMzg0MzMzMzMgOS4wODYsNC4wODEgQzkuNjU1MzMzMzMsMy43Nzc2NjY2NyAxMC4zMzIsMy42MjYgMTEuMTE2LDMuNjI2IEMxMS43NTA2NjY3LDMuNjI2IDEyLjI4OTY2NjcsMy43NDAzMzMzMyAxMi43MzMsMy45NjkgQzEzLjE3NjMzMzMsNC4xOTc2NjY2NyAxMy41OCw0LjUyMiAxMy45NDQsNC45NDIgTDEzLjk1OCwwLjY1OCBDMTMuOTU4LDAuNDUyNjY2NjY3IDE0LjAxNjMzMzMsMC4yOTE2NjY2NjcgMTQuMTMzLDAuMTc1IEMxNC4yNDk2NjY3LDAuMDU4MzMzMzMzMyAxNC40MTA2NjY3LDAgMTQuNjE2LDAgQzE0LjgxMiwwIDE0Ljk2NiwwLjA1ODMzMzMzMzMgMTUuMDc4LDAuMTc1IEMxNS4xOSwwLjI5MTY2NjY2NyAxNS4yNDYsMC40NTI2NjY2NjcgMTUuMjQ2LDAuNjU4IEwxNS4yNDYsMTAuMTA4IEMxNS4yNDYsMTAuMzEzMzMzMyAxNS4xOSwxMC40NzQzMzMzIDE1LjA3OCwxMC41OTEgQzE0Ljk2NiwxMC43MDc2NjY3IDE0LjgxMiwxMC43NjYgMTQuNjE2LDEwLjc2NiBDMTQuNDEwNjY2NywxMC43NjYgMTQuMjQ5NjY2NywxMC43MDc2NjY3IDE0LjEzMywxMC41OTEgQzE0LjAxNjMzMzMsMTAuNDc0MzMzMyAxMy45NTgsMTAuMzEzMzMzMyAxMy45NTgsMTAuMTA4IEwxMy45NTgsOS40MjIgQzEzLjMwNDY2NjcsMTAuMzE4IDEyLjM0OCwxMC43NjYgMTEuMDg4LDEwLjc2NiBaIE0xMS4yMjgsOS42ODggQzEyLjA3NzMzMzMsOS42ODggMTIuNzQ0NjY2Nyw5LjQ2NjMzMzMzIDEzLjIzLDkuMDIzIEMxMy43MTUzMzMzLDguNTc5NjY2NjcgMTMuOTU4LDcuOTcwNjY2NjcgMTMuOTU4LDcuMTk2IEMxMy45NTgsNi40MzA2NjY2NyAxMy43MDEzMzMzLDUuODI0IDEzLjE4OCw1LjM3NiBDMTIuNjc0NjY2Nyw0LjkyOCAxMi4wMjYsNC43MDQgMTEuMjQyLDQuNzA0IEMxMC40Myw0LjcwNCA5Ljc5NTMzMzMzLDQuOTMyNjY2NjcgOS4zMzgsNS4zOSBDOC44ODA2NjY2Nyw1Ljg0NzMzMzMzIDguNjUyLDYuNDU4NjY2NjcgOC42NTIsNy4yMjQgQzguNjUyLDguMDI2NjY2NjcgOC44NzgzMzMzMyw4LjYzOCA5LjMzMSw5LjA1OCBDOS43ODM2NjY2Nyw5LjQ3OCAxMC40MTYsOS42ODggMTEuMjI4LDkuNjg4IFoiIGlkPSJkIiAvPgogICAgICAgICAgICA8L2c+CiAgICAgICAgICAgIDxwYXRoIGQ9Ik00NC45NTIzODEsMTAgTDE5LjA0NzYxOTEsMTAgQzE2LjgxMzMzMzMsMTAgMTUsMTEuODEzMzMzMyAxNSwxNC4wNDc2MTkgTDE1LDM5Ljk1MjM4MSBDMTUsNDIuMTg2NjY2NyAxNi44MTMzMzMzLDQ0IDE5LjA0NzYxOTEsNDQgTDQ0Ljk1MjM4MSw0NCBDNDcuMTg2NjY2Nyw0NCA0OSw0Mi4xODY2NjY3IDQ5LDM5Ljk1MjM4MSBMNDksMTQuMDQ3NjE5IEM0OSwxMS44MTMzMzMzIDQ3LjE4NjY2NjcsMTAgNDQuOTUyMzgxLDEwIFogTTI2Ljc5NDc2MTksMzYuNTY4NTcxNCBMMjUuODcxOTA0OCwzNS4yNDA5NTI0IEwzNy44NjA5NTI0LDI3IEwyNS44NzE5MDQ4LDE4Ljc1OTA0NzYgTDI2Ljc5NDc2MTksMTcuNDMxNDI4NiBMNDAuNzE4NTcxNCwyNyBMMjYuNzk0NzYxOSwzNi41Njg1NzE0IFoiIGlkPSJncmVhdGVyX3RoYW4iIGZpbGw9IiMwMDAwMDAiIC8+CiAgICAgICAgPC9nPgogICAgPC9nPgo8L3N2Zz4=" />
col_sd_gt()
|
|
✓ | 1 | 1 1.00 |
0 0.00 |
-- | -- | -- | -- |