Introduction to Sparvi

Data Profiling Guide

Validation Rules Guide

Anomaly Detection Guide

Sparvi Cloud Overview

Sparvi Cloud Database Connections

## What is Sparvi?

Sparvi is a comprehensive data observability platform that helps you catch data issues early and resolve what impacts business.

### Sparvi Cloud
A full-featured data observability platform for data teams, featuring:
- Interactive data exploration and lineage discovery
- Anomaly detection dashboards with business impact analysis
- Issue management with stakeholder notifications
- Team collaboration and automated monitoring
- Enterprise security with SSO and role-based access

Sparvi Cloud helps data teams maintain high-quality data, prevent issues before they impact business operations, and build confidence in their data assets.

## Core Concepts

### Data Observability

Data observability provides comprehensive visibility into your data's health, quality, and relationships across your organization. Sparvi implements data observability through:

- **Data Profiling**: Automated analysis of data structure, completeness, and statistical patterns
- **Lineage Discovery**: Understanding how data flows through your systems and who depends on it
- **Anomaly Detection**: Identifying unusual patterns and data quality issues with business context
- **Issue Management**: Tracking and resolving data problems with stakeholder awareness
- **Business Impact Analysis**: Understanding which teams and processes are affected by data issues

### Validation Rules

Validation rules enforce business rules and data quality standards. Sparvi Cloud provides:

- **SQL-Based Rules**: Define validation criteria using familiar SQL syntax
- **Health Dashboards**: Visual monitoring of validation performance over time
- **Automated Scheduling**: Run validations on configurable schedules
- **Issue Integration**: Failed validations automatically create tracked issues with business context

### Metadata Management

Sparvi automatically collects and tracks metadata changes across your data infrastructure:

- **Schema Change Detection**: Identify new, modified, or deleted columns
- **Relationship Discovery**: Find dependencies between tables, views, and downstream systems
- **Historical Tracking**: Maintain a complete audit trail of all metadata changes
- **Impact Analysis**: Understand which systems and teams are affected by changes

### Anomaly Detection

Sparvi's anomaly detection identifies data quality issues and provides business context:

- **Configurable Algorithms**: Multiple detection methods tailored to your data patterns
- **Severity Classification**: Automatic prioritization based on business impact
- **Trend Analysis**: Track anomaly patterns over time with visual dashboards
- **Stakeholder Notifications**: Alert the right people when issues affect their work

## Architecture Overview

### Sparvi Cloud Architecture
- **Web Application**: React-based frontend with comprehensive dashboards and workflows
- **API Layer**: RESTful APIs for all platform functionality
- **Database Connectors**: Native integrations with Snowflake (OAuth), PostgreSQL
- **Automation Engine**: Scheduled data profiling, validation runs, and metadata collection
- **Notification System**: Multi-channel alerting with Azure Communication Services
- **Enterprise Security**: Multi-factor authentication, role-based access, user management

## Key Features

- **Data Explorer**: Navigate database schemas, tables, and columns with comprehensive metadata
- **Lineage Discovery**: Automatically map data relationships and business dependencies
- **Issue Management**: Track and resolve data quality problems with business impact context
- **Anomaly Detection Dashboard**: Visual monitoring with configurable detection algorithms
- **Team Collaboration**: User management, notifications, and shared workflows
- **Enterprise Integrations**: Snowflake OAuth, PostgreSQL, automated scheduling

Learn about Sparvi's core concepts and architecture

Data profiling is the process of examining data to understand its structure, content, quality, and patterns. With Sparvi, you can easily profile your database tables to gain deep insights into your data.

## Understanding Sparvi Profiles

When you profile a table with Sparvi, it computes a variety of metrics and statistics that give you a comprehensive view of your data. A profile includes:

### Basic Metrics

- **Row count**: Total number of rows in the table
- **Column count**: Total number of columns
- **Duplicate count**: Number of duplicate rows
- **Timestamp**: When the profile was created

### Completeness Metrics

For each column, Sparvi calculates:

- **Null count**: Number of NULL values
- **Null percentage**: Percentage of NULL values
- **Distinct count**: Number of unique values
- **Distinct percentage**: Percentage of unique values

### Numeric Column Statistics

For numeric columns, Sparvi calculates:

- **Min/Max**: Minimum and maximum values
- **Mean/Median**: Average and middle values
- **Standard Deviation**: Measure of data dispersion
- **Quartiles**: 25th, 50th, and 75th percentiles
- **Outliers**: Values that fall significantly outside the normal range

### Text Column Statistics

For text columns, Sparvi analyzes:

- **Min/Max/Avg Length**: Character length statistics
- **Pattern Recognition**: Common patterns in the data
- **Format Consistency**: Checks for consistent formatting
- **Empty String Detection**: Identifies empty strings (different from NULLs)

### Date Column Statistics

For date and timestamp columns, Sparvi calculates:

- **Min/Max Date**: Earliest and latest dates
- **Date Range**: Span of time represented
- **Temporal Patterns**: Distribution across time periods
- **Future Dates**: Detection of dates in the future
- **Unrealistic Dates**: Identification of dates that are unreasonably old

### Anomaly Detection

Sparvi automatically detects anomalies such as:

- **Statistical Outliers**: Values far from the statistical norm
- **Pattern Deviations**: Inconsistencies in data patterns
- **Distribution Anomalies**: Unusual distributions of values
- **Completeness Issues**: Unexpected NULL patterns
- **Schema Shifts**: Changes in table schema compared to historical data

## Using Data Profiling in Sparvi Cloud

Sparvi Cloud provides automated data profiling through the web interface:

1. **Connect Your Database**: Add your database connection in the Connections section
2. **Navigate to Data Explorer**: Browse your database schemas and tables
3. **View Profiles**: Click on any table to see its comprehensive data profile
4. **Automated Scheduling**: Configure automated profiling schedules for continuous monitoring
5. **Historical Comparison**: Track changes over time with automatic historical comparisons
6. **Anomaly Detection**: Get alerts when unusual patterns or data quality issues are detected

## Analyzing Profiles in Sparvi Cloud

Sparvi Cloud provides comprehensive profile analysis through interactive dashboards and visualizations.

### Viewing Profile Metrics

In the Data Explorer, when you view a table profile, Sparvi Cloud displays:

- **Table Overview**: Row count, column count, duplicate count, and profile timestamp
- **Column Completeness Dashboard**: Visual representation of null percentages and distinct value counts for all columns
- **Interactive Charts**: Sortable and filterable views of all column statistics

### Identifying Data Quality Issues

Sparvi Cloud automatically highlights problematic columns:

- **High Null Rates**: Columns with elevated null percentages are flagged with warning indicators
- **Low Cardinality Issues**: Columns with very low distinct value percentages are identified
- **Sorting and Filtering**: Sort columns by null percentage, distinct count, or other metrics to quickly find issues

### Numeric Column Analysis

For numeric columns, Sparvi Cloud provides:

- **Statistical Summary**: View min, max, average, median, and standard deviation
- **Distribution Visualizations**: Histograms and box plots showing data distribution
- **Outlier Detection**: Automatically identified outliers with example values
- **Trend Analysis**: Historical comparison of numeric statistics over time

### Anomaly Detection

Sparvi Cloud automatically detects and displays anomalies:

- **Anomaly List**: All detected anomalies with severity levels (high, medium, low)
- **Anomaly Details**: Description, affected columns, and detection timestamp
- **Schema Change Tracking**: Automatic detection of schema shifts with before/after comparison
- **Filterable Views**: Filter anomalies by type, severity, or affected table/column

### Dashboard and Reporting

The Sparvi Cloud dashboard provides comprehensive visualizations:

- **Anomaly Trends Over Time**: Line charts showing anomaly detection patterns
- **Severity Breakdown**: Visual distribution of high, medium, and low severity issues
- **Anomaly Type Analysis**: Charts showing which anomaly types are most common
- **Table Health Heatmap**: Color-coded view of data quality across all tables
- **Export Options**: Download reports and share with your team

## Best Practices

1. **Profile Regularly**: Run profiles on a regular schedule to track changes over time
2. **Use Historical Comparisons**: Always compare current profiles with previous ones
3. **Start with Basic Metrics**: Begin analysis with row counts and null percentages
4. **Focus on Outliers**: Pay special attention to statistical outliers
5. **Look for Patterns**: Analyze patterns in text data for inconsistencies
6. **Check Date Ranges**: Ensure date ranges fall within expected bounds
7. **Track Schema Changes**: Monitor changes to table structures over time
8. **Automate Alerting**: Set up alerts for significant changes in key metrics
9. **Adjust Thresholds**: Customize anomaly detection thresholds for your data
10. **Document Findings**: Keep records of data quality issues for future reference

## Next Steps

After profiling your data, consider:

1. Setting up [Validation Rules](/docs/validation-rules) based on profile insights
2. Exploring [Anomaly Detection](/docs/anomaly-detection) for more advanced monitoring
3. Connecting your databases with [Sparvi Cloud Database Connections](/docs/sparvi-cloud-databases)

Learn how to profile your data with Sparvi

Validation rules are the heart of data quality enforcement with Sparvi. They allow you to define specific expectations about your data and verify if those expectations are met.

## Understanding Validation Rules

A validation rule in Sparvi consists of:

1. **Name**: A unique identifier for the rule
2. **Description**: A human-readable explanation of what the rule checks
3. **Query**: The SQL query that performs the validation check
4. **Operator**: How to compare the query result with the expected value
5. **Expected Value**: The value that indicates a passing validation

For example, a rule might check if there are any orders with zero or negative amounts by counting rows where amount is less than or equal to zero. The rule passes if the count equals 0, meaning there are no such orders.

## Validation Rule Anatomy

### Query

The `query` is a SQL statement that returns a single value (typically a COUNT or other aggregate). This value will be compared against the expected value using the specified operator.

Common validation queries include:
- Checking for null values in required columns
- Validating data formats (emails, dates, etc.)
- Ensuring values fall within reasonable date ranges
- Verifying referential integrity between related tables

### Operators

Sparvi supports several comparison operators:

- **equals**: The query result must exactly match the expected value
- **greater_than**: The query result must be greater than the expected value
- **less_than**: The query result must be less than the expected value
- **between**: The query result must fall between two values (requires an array for expected_value)
- **not_equals**: The query result must not match the expected value (opposite of equals)

### Expected Value

The `expected_value` field defines what the query result should be for the validation to pass. The type depends on the operator:

- For `equals`, `greater_than`, `less_than`, and `not_equals`: A single value (number, string, or boolean)
- For `between`: An array of two values `[min, max]`

## Types of Validation Rules

### Data Integrity Rules

These rules check basic integrity constraints:

- **Primary Key Uniqueness**: Ensure all IDs are unique with no duplicates
- **Foreign Key Validity**: Verify that all foreign key references point to valid records in related tables
- **Not Null Constraints**: Check that required fields contain values

### Data Quality Rules

These rules check for data quality issues:

- **Format Validation**: Ensure emails, phone numbers, and other formatted fields follow valid patterns
- **Range Checks**: Verify numeric values fall within reasonable bounds (e.g., ages between 18 and 120)
- **Date Validity**: Confirm dates are not in the future when they shouldn't be
- **Pattern Consistency**: Check that text fields match expected patterns

### Business Logic Rules

These rules enforce business-specific requirements:

- **Conditional Requirements**: Ensure premium users have required information like phone numbers
- **Transaction Verification**: Check that high-value orders or transactions have been properly verified
- **Workflow Validation**: Verify records progress through business processes correctly
- **Relationship Rules**: Ensure related entities maintain expected relationships

### Distribution Rules

These rules check data distributions:

- **Value Distribution**: Ensure no single value dominates the data (e.g., no status represents more than 80% of records)
- **Balance Checks**: Verify data is distributed reasonably across categories
- **Ratio Validation**: Check that proportions between different values remain within expected ranges

## Working with Validation Rules in Sparvi Cloud

### Creating Validation Rules

In Sparvi Cloud, you can create validation rules through the intuitive web interface:

1. **Navigate to Validation Rules**: Access the Validation Rules section from the main navigation
2. **Create New Rule**: Click "Create Rule" to open the rule builder
3. **Define Rule Properties**:
   - Enter a unique name for your rule
   - Write a clear description of what the rule checks
   - Write or select a SQL query that performs the validation
   - Choose the comparison operator (equals, greater_than, less_than, between, not_equals)
   - Set the expected value that indicates a passing validation
4. **Test Your Rule**: Run a test to verify the rule works as expected
5. **Save and Activate**: Save the rule and add it to your validation suite

### Generating Default Rules

Sparvi Cloud can automatically generate sensible validation rules based on your table structure and data patterns:

- Primary key uniqueness checks
- Foreign key validity checks
- Null checks for non-nullable columns
- Format validation for emails, phone numbers, etc.
- Range checks for numeric columns
- Date range validations
- Pattern consistency for text columns

Use the "Generate Rules" feature in the Validation Rules section to create a starting set of rules for your tables.

### Running Validation Rules

In Sparvi Cloud, validation rules can be configured and run through the web interface:

1. **Create Rules**: Define validation rules in the Validation Rules section
2. **Schedule Runs**: Configure automated schedules for your validation rules
3. **Monitor Results**: View validation results in the Health Dashboards
4. **Track Issues**: Failed validations automatically create tracked issues with business context

## Validation Rule Best Practices

### Writing Effective Rules

1. **Use Clear Names**: Rule names should indicate what is being checked
2. **Write Descriptive Explanations**: Descriptions should explain the business reasoning
3. **Keep Queries Simple**: Focus each rule on a single validation check
4. **Validate at the Right Level**: Some checks need row-level detail, others need aggregates
5. **Consider Performance**: Avoid complex joins in validation queries for large tables
6. **Use Parameterized Thresholds**: Set realistic thresholds based on your data volume

### Organizing Validation Rules

Sparvi Cloud helps you organize your rules effectively:

- **Categories and Tags**: Group rules by category (data integrity, data quality, business logic)
- **Table Association**: Link rules to specific tables or schemas
- **Search and Filter**: Quickly find rules using search and filtering
- **Rule Collections**: Create collections of related rules for specific data domains

### Validation Rule Management

As your validation suite grows in Sparvi Cloud:

1. **Rule History**: Track changes to rules with automatic version history
2. **Documentation**: Add notes and documentation directly to each rule
3. **Testing**: Test rules on sample data before activating them in production
4. **Organization**: Group related rules by domain, table, or business function
5. **Prioritization**: Set severity levels (critical, warning, info) for different validations

## Advanced Validation Techniques

### Dynamic Expected Values

Sometimes you need to compare against a dynamic baseline rather than a fixed value. Sparvi Cloud supports:

- **Time-Based Comparisons**: Compare today's metrics against yesterday's or last week's values
- **Relative Thresholds**: Set validation thresholds based on historical averages or trends
- **Adaptive Baselines**: Use calculated values that adjust based on recent data patterns

For example, you can create rules that ensure today's order volume doesn't drop more than 50% compared to yesterday's volume.

### Conditional Validations

Some validations only apply under certain conditions. In Sparvi Cloud you can create:

- **Context-Aware Rules**: Validations that apply only to specific subsets of data
- **If-Then Logic**: Rules that check conditions only when certain criteria are met
- **Segmented Validation**: Different validation thresholds for different data segments

For example, you might validate that international orders have tax IDs, while domestic orders don't require this field.

### Statistical Validations

For data with natural variation, Sparvi Cloud supports statistical approaches:

- **Outlier Detection**: Identify values beyond a certain number of standard deviations from the mean
- **Distribution Checks**: Verify data follows expected statistical distributions
- **Tolerance Ranges**: Set acceptable ranges based on statistical measures rather than fixed values

For example, you can create rules that allow a small number of outliers (e.g., up to 10 values beyond 3 standard deviations) while still flagging when too many extreme values appear.

## Validation Rule Monitoring

Sparvi Cloud provides comprehensive monitoring of validation rule performance:

**Health Dashboards**: Track validation pass rates over time with visual trends

**Failure Tracking**: See which validations are failing most frequently

**Historical Trends**: Understand how data quality evolves over time

**Alert Configuration**: Set up notifications when validation rules start failing

**Team Visibility**: Share validation results with your team through dashboards and issues

## Next Steps

After mastering validation rules:

1. Learn about [Anomaly Detection](/docs/anomaly-detection) to automatically detect data quality issues
2. Explore [Data Profiling](/docs/data-profiling) to understand your data patterns
3. Check out [Sparvi Cloud Overview](/docs/sparvi-cloud-overview) to learn about all platform features

Anomaly detection is a key feature of Sparvi that helps you identify unusual patterns, outliers, and unexpected changes in your data. This guide explains how Sparvi's anomaly detection works and how you can leverage it to maintain high data quality.

## Understanding Anomaly Detection in Sparvi

Anomaly detection in Sparvi works by:

1. **Analyzing current data** to find statistical outliers and pattern deviations
2. **Comparing with historical profiles** to detect changes over time
3. **Applying domain-specific rules** to identify business anomalies
4. **Ranking anomalies by severity** to prioritize the most important issues

Sparvi can detect anomalies in several ways:

- **Automatically** during regular profiling operations
- **Comparatively** when historical data is provided
- **Rule-based** through custom validation rules
- **Time-series based** by analyzing trends and seasonality patterns

## Types of Anomalies

Sparvi detects different types of anomalies:

### Statistical Outliers

Values that deviate significantly from the statistical norm:

- **Numeric Outliers**: Values far from the mean/median (typically beyond 3 standard deviations)
- **Temporal Outliers**: Unusual spikes or drops in time-series data
- **Frequency Outliers**: Unusual distribution of categorical values

### Data Quality Anomalies

Issues that indicate potential data quality problems:

- **Null Rate Changes**: Unexpected increases in null values
- **Duplicate Rate Changes**: Unexpected increases in duplicate records
- **Format Inconsistencies**: Deviations from established data formats
- **Referential Integrity Issues**: Invalid references between tables

### Schema Anomalies

Changes in the structure of your data:

- **Column Additions/Removals**: New or missing columns
- **Data Type Changes**: Changes in column data types
- **Constraint Changes**: Changes in column nullability or uniqueness

### Business Logic Anomalies

Violations of business rules or expectations:

- **Range Violations**: Values outside of acceptable business ranges
- **Process Breaks**: Indications of broken business processes
- **Relationship Anomalies**: Unusual relationships between entities

## Detecting Anomalies with Sparvi Cloud

Sparvi Cloud provides automated anomaly detection through the web interface:

1. **Automatic Detection**: Anomalies are detected automatically during data profiling
2. **Historical Comparison**: Sparvi Cloud tracks changes over time and alerts you to unusual patterns
3. **Anomaly Dashboard**: View all detected anomalies in the Anomaly Detection Dashboard
4. **Configurable Thresholds**: Customize detection sensitivity for different data patterns
5. **Severity Ranking**: Anomalies are automatically ranked by severity to help you prioritize
6. **Business Impact Analysis**: See which teams and processes are affected by detected anomalies
7. **Notifications**: Get alerted via Slack, email, or webhooks when anomalies are detected

## Anomaly Types in Detail

### Row Count Anomalies

Unusual changes in the number of rows in your tables. Sparvi detects when row counts increase or decrease beyond expected thresholds, which could indicate:
- Data pipeline failures
- Missing data loads
- Unexpected data deletions
- Data quality issues upstream

Example: "Row count decreased by 25% (from 1000 to 750)" - marked as high severity

### Null Rate Anomalies

Unexpected changes in null values across your columns. Sparvi monitors null rates and alerts when they deviate from normal patterns, which could indicate:
- Missing data from upstream sources
- Changes in data collection processes
- ETL pipeline issues
- Schema changes affecting data population

Example: "Null rate for 'email' increased from 2% to 15%" - marked as medium severity

### Distribution Anomalies

Changes in the distribution of values within your data. Sparvi detects when value distributions shift significantly, which could indicate:
- Skewed data processing
- Changes in business operations
- Data collection biases
- Incorrect default values

Example: "Unusual distribution of 'status' values (98% are now 'pending')" - marked as high severity

### Statistical Outliers

Values that are statistical outliers compared to historical patterns. Sparvi identifies values that fall outside expected ranges, which could indicate:
- Data entry errors
- System bugs generating invalid values
- Legitimate but unusual business events
- Integration issues with external systems

Example: "Found 5 outlier values in 'amount' (beyond 3 std deviations)" - marked as medium severity

### Format Anomalies

Inconsistencies in data formats across your columns. Sparvi detects when data doesn't match established patterns, which could indicate:
- Changes in data collection processes
- Integration issues with external systems
- Data validation failures
- Migration or import errors

Example: "15% of 'phone_number' values don't match the established pattern" - marked as medium severity

### Schema Shift Anomalies

Changes in your table schema structure. Sparvi monitors for schema changes including:
- New columns added
- Columns removed
- Data type changes
- Constraint modifications

Examples:
- "New column 'discount_code' was added" - marked as medium severity
- "Data type changed from 'DECIMAL(10,2)' to 'INTEGER'" - marked as high severity

## Working with Anomalies in Sparvi Cloud

### Filtering and Viewing Anomalies

In the Anomaly Detection Dashboard, you can:
- Filter anomalies by severity (high, medium, low)
- Filter by anomaly type (row count, null rate, distribution, etc.)
- Filter by affected columns or tables
- View anomalies over specific time periods
- Search for specific anomalies by description

### Taking Action on Anomalies

When Sparvi Cloud detects an anomaly, you can:
1. **Create an Issue**: Convert the anomaly into a tracked issue with business context
2. **Assign to Team Members**: Use @mentions to notify the right people
3. **Add Comments**: Collaborate with your team to investigate and resolve
4. **Track Resolution**: Monitor the status of anomalies from detection to resolution
5. **Document Findings**: Keep a record of what caused the anomaly and how it was fixed

### Anomaly Dashboard

The Anomaly Detection Dashboard provides:
- **Visual Trends**: See anomaly patterns over time
- **Severity Breakdown**: Understand the distribution of high, medium, and low severity anomalies
- **Type Analysis**: View which anomaly types are most common in your data
- **Table Heatmap**: Identify which tables have the most data quality issues
- **Business Impact**: See which teams and processes are affected

## Configuring Anomaly Detection

### Custom Thresholds

In Sparvi Cloud, you can configure detection sensitivity through the settings interface:

**Row Count Changes**: Set the percentage threshold for row count changes (default: 20%)
- Example: Alert only on 30% or greater changes

**Null Rate Changes**: Configure sensitivity for null rate changes (default: 10%)
- Example: Alert on 5% changes for critical columns like email

**Distribution Changes**: Set thresholds for value distribution shifts (default: 20%)
- Example: More sensitive monitoring for status or category columns

### Expected Patterns

Define expected data formats for your columns:
- **Email format**: Standard email validation patterns
- **Phone numbers**: Custom phone number formats for your region
- **Postal codes**: ZIP code or postal code patterns
- **Custom formats**: Define your own regex patterns for specialized data

### Column-Specific Sensitivity

Configure different sensitivity levels for different columns:
- **Critical columns**: Higher sensitivity (e.g., customer email, payment amounts)
- **Reference columns**: Standard sensitivity (e.g., descriptions, notes)
- **Audit columns**: Lower sensitivity (e.g., created_by, updated_at)

## Best Practices

1. **Start Simple**: Begin with default settings and refine thresholds as you learn your data patterns
2. **Historical Comparison**: Sparvi Cloud automatically maintains historical profiles for meaningful anomaly detection
3. **Adjust Thresholds**: Customize thresholds based on your data's natural variability
4. **Column Sensitivity**: Set different sensitivity levels for critical columns vs. less important ones
5. **Severity Triage**: Focus on high-severity anomalies first, then address medium and low ones
6. **Regular Monitoring**: Sparvi Cloud runs profiles on automated schedules to establish normal patterns
7. **Document Findings**: Use issue tracking to keep a record of anomalies and their resolutions
8. **Team Collaboration**: Use @mentions and comments to involve the right people quickly
9. **Pattern Library**: Build a library of expected patterns for format validation
10. **Notification Strategy**: Configure Slack, email, or webhook alerts for critical anomalies

## Next Steps

After mastering anomaly detection:

1. Set up [Validation Rules](/docs/validation-rules) to complement anomaly detection with business-specific checks
2. Explore [Data Profiling](/docs/data-profiling) to understand comprehensive data quality metrics
3. Learn about [Sparvi Cloud Database Connections](/docs/sparvi-cloud-databases) to connect your data sources

Learn how to detect and respond to data anomalies


Sparvi Cloud is a comprehensive data observability platform designed for teams who need collaborative data quality monitoring, issue management, and business impact analysis.

## What Makes Sparvi Cloud Different

Unlike traditional data quality tools that just detect technical issues, Sparvi Cloud provides **business-aware data observability** with automatic stakeholder identification and impact analysis.

### Traditional Approach
```
❌ ORDERS table has 1,247 null customer_ids
Assigned to: @data-engineer
```

### Sparvi Cloud Approach
```
🚨 CRITICAL: ORDERS data quality issue
📊 BUSINESS IMPACT:
  • Sales Dashboard → 12 daily users affected
  • Commission Calculator → Finance team blocked
  • Customer Segmentation → Marketing campaigns paused

👥 AUTO-NOTIFIED:
  • @data-team (technical details)
  • @sales-team (dashboard broken)  
  • @finance-team (payroll delayed)
```

## Core Features

### Data Explorer
Navigate through your database schemas, tables, and columns with comprehensive metadata:
- **Hierarchical Navigation**: Browse databases → schemas → tables → columns
- **Data Preview**: View actual data samples and understand content
- **Metadata Insights**: Row counts, column types, and data patterns
- **Search and Filtering**: Find specific tables and objects quickly

### Data Lineage Discovery
Automatically discover and visualize data relationships:
- **Relationship Mapping**: Understand how data flows through your systems
- **Dependency Analysis**: See what depends on each data source
- **Business Impact**: Identify stakeholders affected by data issues
- **Confidence Scoring**: Know how reliable discovered relationships are

### Anomaly Detection Dashboard
Monitor data quality with intelligent anomaly detection:
- **Configurable Algorithms**: Multiple detection methods for different data patterns
- **Severity Classification**: Automatic prioritization based on business impact
- **Visual Dashboards**: Track trends and patterns over time
- **Threshold Management**: Set appropriate sensitivity levels

### Issue Management System
Track and resolve data quality problems with business context:
- **Automatic Issue Creation**: From failed validations and detected anomalies
- **Business Impact Analysis**: Understand which teams and processes are affected
- **Stakeholder Notifications**: Alert the right people when issues affect their work
- **Resolution Tracking**: Monitor progress and collaborate on fixes

### Custom Validations
Create and manage data quality rules with comprehensive monitoring:
- **SQL-Based Rules**: Use familiar SQL syntax for custom validation logic
- **Health Dashboards**: Visual monitoring of validation performance over time
- **Automated Scheduling**: Run validations on configurable schedules
- **Rule Management**: Organize and maintain validation rule libraries

### Team Collaboration
Enterprise-ready user management and collaboration:
- **Organization Management**: Invite team members and manage access
- **Role-Based Access**: Control who can see and modify different data sources
- **Multi-Factor Authentication**: Secure access with MFA support
- **User Invitations**: Easy onboarding for new team members

## Database Support

Sparvi Cloud provides native integrations with:

### Snowflake
- **OAuth Authentication**: Secure, token-based authentication
- **Key-Pair Authentication**: Support for Snowflake key-pair authentication
- **Warehouse Management**: Specify compute warehouses for different operations
- **Role Support**: Use specific Snowflake roles for data access

### PostgreSQL
- **Standard Authentication**: Username/password connections
- **SSL Support**: Secure connections with SSL encryption
- **Schema Management**: Full support for PostgreSQL schemas and permissions

## Getting Started

### 1. Sign Up and Connect
- Create your Sparvi Cloud account
- Connect your first database (Snowflake or PostgreSQL)
- Configure authentication (OAuth recommended for Snowflake)

### 2. Discover Your Data
- Run automatic data discovery to map your database structure
- Explore tables and understand data patterns
- Review automatically generated metadata

### 3. Set Up Monitoring
- Create validation rules for critical data quality checks
- Configure anomaly detection for key tables
- Set up automated scheduling for regular monitoring

### 4. Invite Your Team
- Add team members to your organization
- Configure roles and permissions
- Set up notification preferences

### 5. Monitor and Improve
- Review data quality dashboards regularly
- Respond to issues with business context
- Continuously refine validation rules and thresholds

## Pricing

Sparvi Cloud offers flexible pricing for teams of all sizes:

- **Free Tier**: 1 user, up to 10 tables monitored
- **Design Partner Program**: Full features with early access and influence on roadmap

For current pricing and features, visit our [pricing page](/#pricing).

## Enterprise Features

### Security and Compliance
- Multi-factor authentication (MFA)
- Role-based access control (RBAC)
- Audit logging and activity tracking
- SOC 2 compliance preparation

### Advanced Integrations
- REST API for custom integrations
- Webhook notifications for external systems
- Single Sign-On (SSO) support (coming soon)
- Custom notification channels

### Dedicated Support
- Priority support response times
- Dedicated customer success manager
- Custom onboarding and training
- Architecture review and optimization

## Next Steps

1. **Try Sparvi Cloud**: [Sign up for early access](/#waitlist)
2. **Schedule a Demo**: See Sparvi Cloud in action with your data
3. **Join Design Partner Program**: Get early access and influence our roadmap
4. **Read Getting Started Guide**: Learn how to connect your first database

## Need Help?

- **Documentation**: Browse our comprehensive guides
- **Support**: Contact our team for technical assistance
- **Community**: Join discussions with other data teams
- **Feedback**: Share your ideas and feature requests

Comprehensive data observability platform for teams


This guide covers how to connect your databases to Sparvi Cloud and configure authentication for secure access.

## Supported Databases

Sparvi Cloud provides native, optimized integrations with:

- **Snowflake** - Username/password and key-pair authentication support

## Snowflake Integration

Sparvi Cloud offers two authentication methods for Snowflake:

### Username/Password Authentication (Recommended)

Standard username and password authentication for Snowflake connections.

**Setup Steps:**
1. **In Sparvi Cloud**: Navigate to Connections → Add New Connection → Snowflake
2. **Choose Username/Password**: Select standard authentication
3. **Configure Connection**: Provide your Snowflake connection details:
   - Account identifier (e.g., `xy12345.us-east-1`)
   - Username
   - Password
   - Database name
   - Schema name (default: `PUBLIC`)
   - Warehouse name
4. **Test Connection**: Verify the connection works correctly

**Benefits:**
- Simple setup process
- Works with existing Snowflake accounts
- No additional configuration required
- Compatible with all Snowflake deployments

### Key-Pair Authentication

For environments where password authentication isn't preferred, Sparvi Cloud supports Snowflake key-pair authentication.

**Setup Steps:**
1. **Generate Key Pair**: Create RSA private/public key pair
2. **Configure Snowflake User**: Add public key to your Snowflake user account
3. **In Sparvi Cloud**: Choose "Key-Pair Authentication" option
4. **Upload Private Key**: Securely upload your private key file
5. **Configure Connection**: Provide connection details and user information
6. **Test Connection**: Verify authentication works

**Security Notes:**
- Private keys are encrypted at rest
- Keys are never logged or exposed
- Follows Snowflake security best practices
- Supports key rotation

### Required Snowflake Permissions

For both authentication methods, your Snowflake user needs these minimum permissions:

```sql
-- Database and schema access
GRANT USAGE ON DATABASE <database_name> TO ROLE <role_name>;
GRANT USAGE ON SCHEMA <database_name>.<schema_name> TO ROLE <role_name>;

-- Table access (for all tables you want to monitor)
GRANT SELECT ON ALL TABLES IN SCHEMA <database_name>.<schema_name> TO ROLE <role_name>;
GRANT SELECT ON FUTURE TABLES IN SCHEMA <database_name>.<schema_name> TO ROLE <role_name>;

-- Information schema access (for metadata discovery)
GRANT USAGE ON DATABASE INFORMATION_SCHEMA TO ROLE <role_name>;

-- Warehouse usage
GRANT USAGE ON WAREHOUSE <warehouse_name> TO ROLE <role_name>;
```


## Connection Security

### Encryption at Rest
- All connection credentials are encrypted using AES-256 encryption
- Encryption keys are managed separately from data
- Regular key rotation following security best practices

### Encryption in Transit
- All database connections use SSL/TLS encryption
- Certificate verification enabled by default
- Support for custom certificates and CA bundles

### Access Controls
- Connection access controlled by organization membership
- Role-based permissions for connection management
- Audit logging for all connection-related activities

### Network Security
- Support for IP whitelisting
- VPC peering available for enterprise customers
- Private endpoint connections (coming soon)

## Connection Management

### Testing Connections
Before saving, Sparvi Cloud tests each connection:
- Verifies authentication credentials
- Checks database permissions
- Tests sample queries
- Validates SSL configuration

### Connection Health Monitoring
- Automatic connection health checks
- Alerts when connections become unavailable
- Connection performance monitoring
- Historical uptime tracking

### Credential Management
- Secure credential storage with encryption
- Support for credential rotation
- Integration with external secret managers (enterprise)
- Audit trail for credential changes

## Troubleshooting

### Common Snowflake Issues

**Username/Password Authentication Failed**
- Double-check username and password are correct
- Verify your Snowflake account identifier format
- Ensure user account is not locked or suspended
- Check that user has permission to use the warehouse

**Key-Pair Authentication Failed**
- Verify private key format (PKCS#8 or PKCS#1)
- Check that public key is correctly associated with user
- Ensure user account is not locked or suspended
- Verify role assignments and permissions

**Connection Timeout**
- Check network connectivity to Snowflake
- Verify warehouse is available and running
- Consider firewall and proxy configurations
- Try connecting from Snowflake web interface


## Best Practices

### Security
- Always use SSL/TLS encryption for production connections
- Implement least-privilege access principles
- Regular credential rotation (quarterly recommended)
- Monitor connection access logs
- Use key-pair authentication for enhanced security (Snowflake)

### Performance
- Choose appropriate Snowflake warehouse sizes
- Configure connection pooling settings
- Monitor query performance and resource usage
- Use efficient sampling strategies for large tables
- Schedule resource-intensive operations during off-peak hours

### Monitoring
- Set up connection health alerts
- Monitor data freshness and availability
- Track query performance metrics
- Review access patterns and usage
- Regular permission audits

## Enterprise Features

### Advanced Authentication
- Single Sign-On (SSO) integration (coming soon)
- LDAP/Active Directory integration
- Multi-factor authentication enforcement
- Custom authentication providers

### Network Security
- VPC peering for private connections
- IP whitelisting and access controls
- Private endpoints and service connections
- Custom DNS and network routing

### Compliance
- SOC 2 Type II certification
- GDPR compliance features
- Data residency options
- Audit logging and reporting

## Next Steps

1. **Connect Your Database**: Start with our connection wizard
2. **Verify Permissions**: Ensure your database user has required access
3. **Test Data Discovery**: Run initial metadata discovery
4. **Set Up Monitoring**: Configure validation rules and anomaly detection
5. **Invite Team Members**: Add colleagues and configure roles

Need help connecting your database? [Contact our support team](/#contact) for personalized assistance.

How to connect Snowflake databases to Sparvi Cloud

Documentation