# Schema Validation Schema validation in Pointblank allows you to verify that your data conforms to an expected structure and type specification. This is particularly useful when ensuring data consistency across systems or validating incoming data against predefined requirements. Let's first look at the dataset we'll use for the first example: ``` python import pointblank as pb # Preview the small_table dataset we'll use throughout this guide pb.preview(pb.load_dataset(dataset="small_table", tbl_type="polars")) ```
PolarsRows13Columns8
date_time Datetime
date Date
a Int64
b String
c Int64
d Float64
e Boolean
f String
1 2016-01-04 11:00:00 2016-01-04 2 1-bcd-345 3 3423.29 True high
2 2016-01-04 00:32:00 2016-01-04 3 5-egh-163 8 9999.99 True low
3 2016-01-05 13:32:00 2016-01-05 6 8-kdg-938 3 2343.23 True high
4 2016-01-06 17:23:00 2016-01-06 2 5-jdo-903 None 3892.4 False mid
5 2016-01-09 12:36:00 2016-01-09 8 3-ldm-038 7 283.94 True low
9 2016-01-20 04:30:00 2016-01-20 3 5-bce-642 9 837.93 False high
10 2016-01-20 04:30:00 2016-01-20 3 5-bce-642 9 837.93 False high
11 2016-01-26 20:07:00 2016-01-26 4 2-dmx-010 7 833.98 True low
12 2016-01-28 02:51:00 2016-01-28 2 7-dmx-010 8 108.34 False low
13 2016-01-30 11:23:00 2016-01-30 1 3-dka-303 None 2230.09 True high
# Schema Definition and Validation A schema in Pointblank is created using the [Schema](../../reference/Schema.md#pointblank.Schema) class which defines the expected structure of a table. Once created, you apply schema validation through the Validate.col_schema_match() validation step. ``` python # Create a schema definition matching small_table structure schema = pb.Schema( columns=[ ("date_time",), # Only check column name ("date",), # Only check column name ("a", "Int64"), # Check name and type ("b", "String"), # Check name and type ("c", "Int64"), # Check name and type ("d", "Float64"), # Check name and type ("e", "Boolean"), # Check name and type ("f",), # Only check column name ] ) # Validate the small_table against the schema small_table_validation = ( pb.Validate( data=pb.load_dataset(dataset="small_table", tbl_type="polars"), label="Schema validation of `small_table`.", ) .col_schema_match(schema=schema) .interrogate() ) small_table_validation ```
Pointblank Validation
Schema validation of `small_table`. Polars
STEP COLUMNS VALUES TBL EVAL UNITS PASS FAIL W E C EXT
#4CA64C 1
col_schema_match()
-- SCHEMA ✓ 1 1
1.00		 0
0.00		 -- -- -- --
Notes

Step 1 (schema_check) ✓ Schema validation passed.

Schema Comparison
TARGET EXPECTED
COLUMN DATA TYPE COLUMN DATA TYPE
1 date_time Datetime(time_unit='us', time_zone=None) 1 date_time ✓ --
2 date Date 2 date ✓ --
3 a Int64 3 a ✓ Int64 ✓
4 b String 4 b ✓ String ✓
5 c Int64 5 c ✓ Int64 ✓
6 d Float64 6 d ✓ Float64 ✓
7 e Boolean 7 e ✓ Boolean ✓
8 f String 8 f ✓ --
Supplied Column Schema: [('date_time',), ('date',), ('a', 'Int64'), ('b', 'String'), ('c', 'Int64'), ('d', 'Float64'), ('e', 'Boolean'), ('f',)]
Schema Match Settings COMPLETE IN ORDER COLUMN ≠ column DTYPE ≠ dtype float ≠ float64
The output shows the validation passed successfully. When all columns have the correct names and types as specified in the schema, the validation passes with a single passing test unit. If there were discrepancies, this would fail, but the basic output wouldn't show specific issues. For detailed information about validation results, use Validate.get_step_report(): ``` python small_table_validation.get_step_report(i=1) ```
Report for Validation Step 1 ✓ COLUMN SCHEMA MATCH COMPLETE IN ORDER COLUMN ≠ column DTYPE ≠ dtype float ≠ float64
TARGET EXPECTED
COLUMN DATA TYPE COLUMN DATA TYPE
1 date_time Datetime(time_unit='us', time_zone=None) 1 date_time ✓ --
2 date Date 2 date ✓ --
3 a Int64 3 a ✓ Int64 ✓
4 b String 4 b ✓ String ✓
5 c Int64 5 c ✓ Int64 ✓
6 d Float64 6 d ✓ Float64 ✓
7 e Boolean 7 e ✓ Boolean ✓
8 f String 8 f ✓ --
Supplied Column Schema: [('date_time',), ('date',), ('a', 'Int64'), ('b', 'String'), ('c', 'Int64'), ('d', 'Float64'), ('e', 'Boolean'), ('f',)]
The step report provides specific details about which columns were checked and whether they matched the schema, helping diagnose issues when validation fails. # Schema Components and Column Types When defining a schema, you need to specify column names and optionally their data types. By default, Pointblank enforces strict validation where: - all columns in your table must match the specified schema - column order must match the schema - column types are case-sensitive - type names must match exactly The schema definition accepts column types as string representations, which vary depending on your data source: - `string`: Character data (may also be `"String"`, `"varchar"`, `"character"`, etc.) - `integer`: Integer values (may also be `"Int64"`, `"int"`, `"bigint"`, etc.) - `numeric`: Numeric values including integers and floating-point numbers (may also be `"Float64"`, `"double"`, `"decimal"`, etc.) - `boolean`: Logical values (`True`/`False`) (may also be `"Boolean"`, `"bool"`, etc.) - `datetime`: Date and time values (may also be `"Datetime"`, `"timestamp"`, etc.) - `date`: Date values (may also be `"Date"`, etc.) - `time`: Time values For specific database engines or DataFrame libraries, you may need to use their exact type names (like `"VARCHAR(255)"` for SQL databases or `"Int64"` for Polars integers). # Discovering Column Types To easily determine the correct type string for columns in your data, Pointblank provides two helpful functions: ``` python import polars as pl from datetime import date # Define a sample dataframe sample_df = pl.DataFrame({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"], "join_date": [date(2020, 1, 1), date(2021, 3, 15), date(2022, 7, 10)] }) ``` ``` python # Method 1: Using `preview()` with `show_types=True` to see column types pb.preview(sample_df) ```
PolarsRows3Columns3
id Int64
name String
join_date Date
1 1 Alice 2020-01-01
2 2 Bob 2021-03-15
3 3 Charlie 2022-07-10
``` python # Method 2: Using `col_summary_tbl()` which shows column types and other details pb.col_summary_tbl(sample_df) ```
PolarsRows3Columns3
Column NA UQ Mean SD Min P5 Q1 Med Q3 P95 Max IQR
id Int64
0
0		 3
1		 2 1 1 1.01 1.5 2 2.5 2.9 3 1
name String
0
0		 3
1		 5 2 3 3.02 4 5 6 6.8 7 2
join_date Date
0
0		 3
1		 - - 2020
01
01		 - - - - - 2022
07
10		 -
String columns statistics regard the string's length.
These functions help you identify the exact type strings to use in your schema definitions, eliminating guesswork and ensuring compatibility with your data source. # Creating a Schema You can create a schema in four different ways, each with its own advantages. All schema objects can be printed to display their column names and data types. ## 1. Using a List of Tuples with `columns=` This approach allows for mixed validation: some columns checked for both name and type, others only for name: ``` python schema_tuples = pb.Schema( # List of tuples approach: flexible for mixed type/name checking --- columns=[ ("name", "String"), # Check name and type ("age", "Int64"), # Check name and type ("height",) # Check name only ] ) print(schema_tuples) ``` Pointblank Schema name: String age: Int64 height: This is the only method that allows checking just column names for some columns while checking both names and types for others. ## 2. Using a Dictionary with `columns=` This approach is often the most readable when defining a schema manually, especially for larger schemas: ``` python schema_dict = pb.Schema( # Dictionary approach (keys are column names, values are data types) --- columns={ "name": "String", "age": "Int64", "height": "Float64", "created_at": "Datetime" } ) print(schema_dict) ``` Pointblank Schema name: String age: Int64 height: Float64 created_at: Datetime With this method, you must always provide both column names (as keys) and their types (as values). ## 3. Using Keyword Arguments For more readable code with a small number of columns: ``` python schema_kwargs = pb.Schema( # Keyword arguments approach (more readable for simple schemas) --- name="String", age="Int64", height="Float64" ) print(schema_kwargs) ``` Pointblank Schema name: String age: Int64 height: Float64 Like the dictionary method, this approach requires both column names and types. ## 4. Extracting from an Existing Table with `tbl=` You can automatically extract a schema from an existing table: ``` python import polars as pl # Create a sample dataframe df = pl.DataFrame({ "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35], "height": [5.6, 6.0, 5.8] }) # Extract schema from table schema_from_table = pb.Schema(tbl=df) print(schema_from_table) ``` Pointblank Schema name: String age: Int64 height: Float64 This is especially useful when you want to validate that future data matches the structure of a reference dataset. # Multiple Data Types for a Column You can specify multiple acceptable types for a column by providing a list of types: ``` python # Schema with multiple possible types for a column schema_multi_types = pb.Schema( columns={ "name": "String", "age": ["Int64", "Float64"], # Accept either integer or float "active": "Boolean" } ) print(schema_multi_types) ``` Pointblank Schema name: String age: ['Int64', 'Float64'] active: Boolean This is useful when working with data sources that might represent the same information in different ways (e.g., integers sometimes stored as floats). # Schema Validation Options When using [col_schema_match()](../../reference/Validate.col_schema_match.md#pointblank.Validate.col_schema_match), you can customize validation behavior with several important options: | Option | Default | Description | |----|----|----| | `complete` | `True` | Require exact column presence (no extra columns allowed) | | `in_order` | `True` | Enforce column order | | `case_sensitive_colnames` | `True` | Make column name matching case-sensitive | | `case_sensitive_dtypes` | `True` | Make data type matching case-sensitive | | `full_match_dtypes` | `True` | Require exact (not partial) type name matches | ## Controlling Column Presence By default, Validate.col_schema_match() requires a complete match between the schema's columns and the table's columns. You can make this more flexible: ``` python # Create a sample table users_table_extra = pl.DataFrame({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35], "extra_col": ["a", "b", "c"] # Extra column not in schema }) # Create a schema schema = pb.Schema( columns={"id": "Int64", "name": "String", "age": "Int64"} ) # Validate without requiring all columns to be present validation = ( pb.Validate(data=users_table_extra) .col_schema_match( schema=schema, # Allow schema columns to be a subset --- complete=False ) .interrogate() ) validation.get_step_report(i=1) ```
Report for Validation Step 1 ✓ COLUMN SCHEMA MATCH COMPLETE IN ORDER COLUMN ≠ column DTYPE ≠ dtype float ≠ float64
TARGET EXPECTED
COLUMN DATA TYPE COLUMN DATA TYPE
1 id Int64 1 id ✓ Int64 ✓
2 name String 2 name ✓ String ✓
3 age Int64 3 age ✓ Int64 ✓
4 extra_col String
Supplied Column Schema: [('id', 'Int64'), ('name', 'String'), ('age', 'Int64')]
## Column Order Enforcement You can control whether column order matters in your validation: ``` python # Create a sample table users_table = pl.DataFrame({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35], }) # Create a schema schema = pb.Schema( columns={"name": "String", "age": "Int64", "id": "Int64"} ) # Validate without enforcing column order validation = ( pb.Validate(data=users_table) .col_schema_match( schema=schema, # Don't enforce column order --- in_order=False ) .interrogate() ) validation.get_step_report(i=1) ```
Report for Validation Step 1 ✓ COLUMN SCHEMA MATCH COMPLETE IN ORDER COLUMN ≠ column DTYPE ≠ dtype float ≠ float64
TARGET EXPECTED
COLUMN DATA TYPE COLUMN DATA TYPE
1 id Int64 3 id ✓ Int64 ✓
2 name String 1 name ✓ String ✓
3 age Int64 2 age ✓ Int64 ✓
Supplied Column Schema: [('name', 'String'), ('age', 'Int64'), ('id', 'Int64')]
## Case Sensitivity Control whether column names and data types are case-sensitive: ``` python # Create schema with different case charactistics case_schema = pb.Schema( columns={"ID": "int64", "NAME": "string", "AGE": "int64"} ) # Create validation with case-insensitive column names and types validation = ( pb.Validate(data=users_table) .col_schema_match( schema=case_schema, # Ignore case in column names --- case_sensitive_colnames=False, # Ignore case in data type names --- case_sensitive_dtypes=False ) .interrogate() ) validation.get_step_report(i=1) ```
Report for Validation Step 1 ✓ COLUMN SCHEMA MATCH COMPLETE IN ORDER COLUMN = column DTYPE = dtype float ≠ float64
TARGET EXPECTED
COLUMN DATA TYPE COLUMN DATA TYPE
1 id Int64 1 ID ✓ int64 ✓
2 name String 2 NAME ✓ string ✓
3 age Int64 3 AGE ✓ int64 ✓
Supplied Column Schema: [('ID', 'int64'), ('NAME', 'string'), ('AGE', 'int64')]
## Type Matching Precision Control how strictly data types must match: ``` python # Create schema with simplified type names type_schema = pb.Schema( # Using simplified type names --- columns={"id": "int", "name": "str", "age": "int"} ) # Allow partial type matches validation = ( pb.Validate(data=users_table) .col_schema_match( schema=type_schema, # Ignore case in data type names --- case_sensitive_dtypes=False, # Allow partial type name matches --- full_match_dtypes=False ) .interrogate() ) validation.get_step_report(i=1) ```
Report for Validation Step 1 ✓ COLUMN SCHEMA MATCH COMPLETE IN ORDER COLUMN ≠ column DTYPE = dtype float = float64
TARGET EXPECTED
COLUMN DATA TYPE COLUMN DATA TYPE
1 id Int64 1 id ✓ int ✓
2 name String 2 name ✓ str ✓
3 age Int64 3 age ✓ int ✓
Supplied Column Schema: [('id', 'int'), ('name', 'str'), ('age', 'int')]
# Common Schema Validation Patterns This section explores common patterns for applying schema validation to different scenarios. Each pattern addresses specific validation needs you might encounter when working with real-world data. We'll examine the step reports (Validate.get_step_report()) for these validations since they provide more detailed information about what was checked and how the validation performed, offering an intuitive way to understand the results beyond simple pass/fail indicators. ## Structural Validation Only When you only care about column names but not their types: ``` python # Create a schema with only column names structure_schema = pb.Schema( columns=["id", "name", "age", "extra_col"] ) # Validate structure only validation = ( pb.Validate(data=users_table_extra) .col_schema_match(schema=structure_schema) .interrogate() ) validation.get_step_report(i=1) ```
Report for Validation Step 1 ✓ COLUMN SCHEMA MATCH COMPLETE IN ORDER COLUMN ≠ column DTYPE ≠ dtype float ≠ float64
TARGET EXPECTED
COLUMN DATA TYPE COLUMN DATA TYPE
1 id Int64 1 id ✓ --
2 name String 2 name ✓ --
3 age Int64 3 age ✓ --
4 extra_col String 4 extra_col ✓ --
Supplied Column Schema: [('id',), ('name',), ('age',), ('extra_col',)]
## Mixed Validation Validate types for critical columns but just presence for others: ``` python # Mixed validation for different columns mixed_schema = pb.Schema( columns=[ ("id", "Int64"), # Check name and type ("name", "String"), # Check name and type ("age",), # Check name only ("extra_col",) # Check name only ] ) # Validate with mixed approach validation = ( pb.Validate(data=users_table_extra) .col_schema_match(schema=mixed_schema) .interrogate() ) validation.get_step_report(i=1) ```
Report for Validation Step 1 ✓ COLUMN SCHEMA MATCH COMPLETE IN ORDER COLUMN ≠ column DTYPE ≠ dtype float ≠ float64
TARGET EXPECTED
COLUMN DATA TYPE COLUMN DATA TYPE
1 id Int64 1 id ✓ Int64 ✓
2 name String 2 name ✓ String ✓
3 age Int64 3 age ✓ --
4 extra_col String 4 extra_col ✓ --
Supplied Column Schema: [('id', 'Int64'), ('name', 'String'), ('age',), ('extra_col',)]
## Progressive Schema Evolution As your data evolves, you might need to adapt your validation approach: ``` python # Original schema original_schema = pb.Schema( columns={ "id": "Int64", "name": "String" } ) # New data with additional columns evolved_data = pl.DataFrame({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35], # New column "active": [True, False, True] # New column }) # Validate with flexible parameters validation = ( pb.Validate(evolved_data) .col_schema_match( schema=original_schema, # Allow extra columns --- complete=False, # Don't enforce order --- in_order=False ) .interrogate() ) validation.get_step_report(i=1) ```
Report for Validation Step 1 ✓ COLUMN SCHEMA MATCH COMPLETE IN ORDER COLUMN ≠ column DTYPE ≠ dtype float ≠ float64
TARGET EXPECTED
COLUMN DATA TYPE COLUMN DATA TYPE
1 id Int64 1 id ✓ Int64 ✓
2 name String 2 name ✓ String ✓
3 age Int64
4 active Boolean
Supplied Column Schema: [('id', 'Int64'), ('name', 'String')]
# Integrating with Larger Validation Workflows Schema validation is often just one part of a comprehensive data validation strategy. You can combine schema checks with other validation steps: ``` python # Define a schema schema = pb.Schema( columns={ "id": "Int64", "name": "String", "age": "Int64" } ) # Create a validation plan validation = ( pb.Validate( users_table, label="User data validation", thresholds=pb.Thresholds(warning=0.05, error=0.1) ) # Add schema validation --- .col_schema_match(schema=schema) # Add other validation steps --- .col_vals_not_null(columns="id") .col_vals_gt(columns="age", value=26) .interrogate() ) validation ``` NDMzMTU2LDEyLjQxOTgzODMgTDEzLjYwNTY5NTYsMTQuOTY2NDkzIEMxMy44OTE2OTcsMTUuMTgwMzcyNSAxNC4yOTcwNzI5LDE1LjEyMzE3MjEgMTQuNTEwOTUxNywxNC44MzcxNzA3IEMxNC43MjQ4MzEzLDE0LjU1MTE2OTIgMTQuNjY3NjMwOSwxNC4xNDU3OTQgMTQuMzgxNjI5NCwxMy45MzE5MTQ1IEwxMi41MzEzMjU3LDEyLjUzOTIxMjcgTDIxLjg4MTI0OTUsMTIuNTM5MjEyNyBMMjEuODgxMjQ5NSwxMS4yNjU4ODU0IEwxMi41MzEzMjU3LDExLjI2NTg4NTQgTDE0LjM4MTYyOTQsOS44NzMxODM2NCBDMTQuNjM3Nzg3Miw5LjcxNjUwNDUzIDE0Ljc0OTcwMDYsOS40MDA2NjAxNCAxNC42NDc3MzUxLDkuMTE3MTQ1NTMgQzE0LjU0ODI1NjQsOC44MzM2MzE1NiAxNC4yNjIyNTUsOC42NTk1NDM1MiAxMy45NjM4MTg5LDguNjk5MzM1IFoiIGlkPSJhcnJvdyIgZmlsbD0iIzAwMDAwMCIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMTUuOTI5MjMwLCAxMS44OTQ3MzcpIHJvdGF0ZSgtMTgwLjAwMDAwMCkgdHJhbnNsYXRlKC0xNS45MjkyMzAsIC0xMS44OTQ3MzcpICIgLz4KICAgICAgICA8L2c+CiAgICA8L2c+Cjwvc3ZnPg==" />
Pointblank Validation
User data validation PolarsWARNING0.05ERROR0.1CRITICAL--
STEP COLUMNS VALUES TBL EVAL UNITS PASS FAIL W E C EXT
#4CA64C 1
AuNzEyMjM0LDEuMDE0NjY5MzUgWiIgaWQ9InJlY3RhbmdsZSIgc3Ryb2tlPSIjMDAwMDAwIiBzdHJva2Utd2lkdGg9IjIiIGZpbGw9IiNGRkZGRkYiIC8+CiAgICAgICAgICAgIDxwYXRoIGQ9Ik01My43MTIyMzQsMzkuNzg4NTI2OCBMNTQuMjEyMjM0LDU2LjI4ODUyNjggTDQyLjIxMjIzNCw1Ni43ODg1MjY4IEw0Mi4yMTIyMzQsMzkuNzg4NTI2OCBMNTMuNzEyMjM0LDM5Ljc4ODUyNjggWiBNMzkuNzEyMjM0LDM5Ljc4ODUyNjggTDM5LjcxMjIzNCw1Ni43ODg1MjY4IEwyNy43MTIyMzQsNTYuNzg4NTI2OCBMMjcuNzEyMjM0LDM5Ljc4ODUyNjggTDM5LjcxMjIzNCwzOS43ODg1MjY4IFogTTI1LjIxMjIzNCwzOS43ODg1MjY4IEwyNS4yMTIyMzQsNTYuNzg4NTI2OCBMMTMuNzEyMjM0LDU2Ljc4ODUyNjggTDEzLjIxMjIzNCw0MC4yODg1MjY4IEwyNS4yMTIyMzQsMzkuNzg4NTI2OCBaIiBpZD0iY29sdW1uc19zY2hlbWEiIHN0cm9rZT0iIzAwMDAwMCIgZmlsbC1ydWxlPSJub256ZXJvIiAvPgogICAgICAgICAgICA8ZyBpZD0idmVydGljYWxfZXF1YWwiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDMwLjAwMDAwMCwgMjkuMDAwMDAwKSIgc3Ryb2tlPSIjMDAwMDAwIiBzdHJva2UtbGluZWNhcD0ic3F1YXJlIj4KICAgICAgICAgICAgICAgIDxsaW5lIHgxPSIyLjIxMjIzMzk3IiB5MT0iMC41MTQ2NjkzNTMiIHgyPSIyLjIxMjIzMzk3IiB5Mj0iNy41ODU3MzcxNiIgaWQ9IkxpbmUiPjwvbGluZT4KICAgICAgICAgICAgICAgIDxsaW5lIHgxPSI1LjIxMjIzMzk3IiB5MT0iMC41MTQ2NjkzNTMiIHgyPSI1LjIxMjIzMzk3IiB5Mj0iNy41ODU3MzcxNiIgaWQ9IkxpbmUtQ29weSI+PC9saW5lPgogICAgICAgICAgICA8L2c+CiAgICAgICAgICAgIDxwYXRoIGQ9Ik00MS43MTIyMzQsOS4wMTQ2NjkzNSBMNDEuNzEyMjM0LDI3LjAxNDY2OTQgTDUzLjcxMjIzNCwyNy4wMTQ2Njk0IEM1NC4yNjIyMzQsMjcuMDE0NjY5NCA1NC43MTIyMzQsMjYuNTY0NjY5NCA1NC43MTIyMzQsMjYuMDE0NjY5NCBMNTQuNzEyMjM0LDEwLjAxNDY2OTQgQzU0LjcxMjIzNCw5LjQ2NDY2OTM1IDU0LjI2MjIzNCw5LjAxNDY2OTM1IDUzLjcxMjIzNCw5LjAxNDY2OTM1IEw0MS43MTIyMzQsOS4wMTQ2NjkzNSBaIE0yNy4yMTIyMzQsOS4wMTQ2NjkzNSBDMjcuMjEyMjM0LDkuMDE0NjY5MzUgMjcuMjEyMjM0LDE1LjAxNDY2OTQgMjcuMjEyMjM0LDI3LjAxNDY2OTQgTDQwLjIxMjIzNCwyNy4wMTQ2Njk0IEw0MC4yMTIyMzQsOS4wMTQ2NjkzNSBDMzEuNTQ1NTY3Myw5LjAxNDY2OTM1IDI3LjIxMjIzNCw5LjAxNDY2OTM1IDI3LjIxMjIzNCw5LjAxNDY2OTM1IFogTTEzLjcxMjIzNCw5LjAxNDY2OTM1IEMxMy4xNjIyMzQsOS4wMTQ2NjkzNSAxMi43MTIyMzQsOS40NjQ2NjkzNSAxMi43MTIyMzQsMTAuMDE0NjY5NCBMMTIuNzEyMjM0LDI2LjAxNDY2OTQgQzEyLjcxMjIzNCwyNi41NjQ2Njk0IDEzLjE2MjIzNCwyNy4wMTQ2Njk0IDEzLjcxMjIzNCwyNy4wMTQ2Njk0IEwyNS43MTIyMzQsMjcuMDE0NjY5NCBMMjUuNzEyMjM0LDkuMDE0NjY5MzUgTDEzLjcxMjIzNCw5LjAxNDY2OTM1IFoiIGlkPSJjb2x1bW5zX3JlYWwiIGZpbGw9IiMwMDAwMDAiIGZpbGwtcnVsZT0ibm9uemVybyIgLz4KICAgICAgICA8L2c+CiAgICA8L2c+Cjwvc3ZnPg==" /> col_schema_match()
-- SCHEMA ✓ 1 1
1.00		 0
0.00		 ○ ○ -- --
#4CA64C 2
col_vals_not_null()
id -- ✓ 3 3
1.00		 0
0.00		 ○ ○ -- --
#EBBC14 3
col_vals_gt()
age 26 ✓ 3 2
0.67		 1
0.33		 ● ● -- CSV
Notes

Step 1 (schema_check) ✓ Schema validation passed.

Schema Comparison
TARGET EXPECTED
COLUMN DATA TYPE COLUMN DATA TYPE
1 id Int64 1 id ✓ Int64 ✓
2 name String 2 name ✓ String ✓
3 age Int64 3 age ✓ Int64 ✓
Supplied Column Schema: [('id', 'Int64'), ('name', 'String'), ('age', 'Int64')]
Schema Match Settings COMPLETE IN ORDER COLUMN ≠ column DTYPE ≠ dtype float ≠ float64
This approach allows you to first validate the structure of your data and then check specific business rules or constraints. # Best Practices 1. Define schemas early: document and define expected data structures early in your data workflow. 2. Choose the right creation method: - use `columns=` for readability with many columns - use `columns=` for mixed name/type validation - use `kwargs` for small schemas with simple column names - use `tbl=` to extract schemas from reference datasets 3. Be deliberate about strictness: choose validation parameters based on your specific needs: - strict validation (`complete=True`) for critical data interfaces - flexible validation (`complete=False`, `in_order=False`) for evolving datasets 4. Reuse schemas: create schema definitions that can be reused across multiple validation contexts. 5. Version control schemas: as your data evolves, maintain versions of your schemas to track changes. 6. Extract schemas from reference data: when you have a 'golden' dataset that represents your ideal structure, use `Schema(tbl=reference_data)` to extract its schema. 7. Consider type flexibility: use multiple types per column (`["Int64", "Float64"]`) when working with data from diverse sources. 8. Combine with targeted validation: use schema validation for structural checks and add specific validation steps for business rules. # Conclusion Schema validation provides a powerful mechanism for ensuring your data adheres to expected structural requirements. It serves as an excellent first line of defense in your data validation strategy, verifying that the data you're working with has the expected shape before applying more detailed business rule validations. The [Schema](../../reference/Schema.md#pointblank.Schema) class offers multiple ways to define schemas, from manual specification with dictionaries or keyword arguments to automatic extraction from reference tables. When combined with the flexible options of Validate.col_schema_match(), you can implement validation approaches ranging from strict structural enforcement to more flexible evolution-friendly checks. By understanding the different schema creation methods and validation options, you can efficiently validate the structure of your data tables and ensure they meet your requirements before processing.