Data Quality
Understanding the critical role of data quality in building reliable AI/ML systems.
What is Data Quality?
Data Quality refers to the measure of how well data serves its intended purpose. In AI and machine learning, high-quality data is essential for training accurate models, reducing bias, and ensuring reliable predictions. Poor data quality is one of the primary reasons why AI projects fail to achieve their goals.
⚠️ Key Insight: "Garbage In, Garbage Out" — Studies show that data scientists spend 60-80% of their time on data cleaning and preparation, making data quality the foundation of every successful AI project.
📊 Core Quality Dimensions
✅ Accuracy
Data correctly represents real-world values and facts.
📋 Completeness
No missing values or gaps in required fields.
🔄 Consistency
Uniform format and values across all data sources.
⏱️ Timeliness
Data is current and available when needed.
🔍 Validity
Data conforms to defined formats and business rules.
🔗 Uniqueness
No duplicate records or redundant entries.
🤖 AI-Specific Quality Dimensions
Beyond traditional dimensions, AI/ML data requires additional quality considerations:
Representativeness & Balance
Training data should reflect real-world distributions. Imbalanced classes lead to biased models that underperform on minority categories.
Label Quality
For supervised learning, accurate and consistent labeling is crucial. Noisy labels degrade model performance significantly.
Bias & Toxicity Detection
Identify and mitigate harmful patterns, stereotypes, or toxic content that could be learned by the model.
Feature Quality
Features should be informative, non-redundant, and have appropriate scales and distributions for the model.
🔧 Data Quality Pipeline
💻 Data Quality Checks with Great Expectations
import great_expectations as gx
# Create a data context
context = gx.get_context()
# Connect to your data
datasource = context.sources.add_pandas("my_datasource")
data_asset = datasource.add_dataframe_asset("customer_data")
# Define expectations (quality rules)
expectation_suite = context.add_expectation_suite("customer_quality")
# Add quality checks
expectation_suite.add_expectation(
gx.expectations.ExpectColumnValuesToNotBeNull(column="email")
)
expectation_suite.add_expectation(
gx.expectations.ExpectColumnValuesToMatchRegex(
column="email",
regex=r"^[\w\.-]+@[\w\.-]+\.\w+$"
)
)
expectation_suite.add_expectation(
gx.expectations.ExpectColumnValuesToBeBetween(
column="age", min_value=0, max_value=120
)
)
# Run validation
results = context.run_checkpoint(checkpoint_name="quality_check")
print(f"Validation passed: {results.success}")🦾 LLM-Specific Data Quality
For Large Language Models, additional quality considerations apply:
📄 Text Quality
- • Grammar and spelling correctness
- • Coherence and readability
- • Language detection and filtering
🔄 Deduplication
- • Exact match deduplication
- • Near-duplicate detection (MinHash)
- • Semantic deduplication
🔐 PII Removal
- • Named entity recognition
- • Regex patterns for emails, phones
- • Data anonymization/masking
⚡ Content Filtering
- • Toxicity and hate speech detection
- • NSFW content removal
- • Quality scoring (perplexity)
📈 Quality Metrics Dashboard
| Metric | Description | Target |
|---|---|---|
| Null Rate | % of missing values per column | < 5% |
| Duplicate Rate | % of duplicate records | < 1% |
| Schema Violations | Records failing type/format checks | 0% |
| Label Agreement | Inter-annotator agreement score | > 85% |
| Data Freshness | Time since last data update | Depends |
🛠️ Popular Tools
Data validation
Data testing
Data profiling
Label quality