LogoLogo
  • Datafold
  • Introduction
    • Data Diff
      • Continuous Integration
      • Manual Data Diff
      • Diff Results
    • Column-level lineage
      • Usage, popularity, & impact per table or column
    • Alerting
  • ⏱️Quickstart Guide
  • Getting Started
    • Data Warehouses
      • Snowflake
      • BigQuery
      • Redshift
      • Postgres
      • Databricks
    • Configuration
      • Indexing
      • Filtering
      • Profiling
      • Lineage
    • On-prem Deployment
      • AWS
      • GCP
    • SSO
      • Okta
      • Google OAuth
      • SAML
  • Integrations
    • Continuous Integration
      • Source Control with Git
        • GitHub
          • On-prem Github
        • GitLab
      • dbt Cloud
      • dbt Core / datafold-sdk
        • GitHub example
        • GitLab example
      • dbt Configurations
      • datafold-sdk
    • Alert Integrations
      • Slack integration
        • Slack Alerts
        • On-prem Slack Integration
      • Alerting webhooks
    • Data Apps
      • Mode
      • Hightouch
  • Developer
    • Datafold API
      • Alerting
      • GraphQL Metadata API
      • Data Diff
      • Error handling
    • Security
      • GDPR
      • Network Security
Powered by GitBook
On this page

Was this helpful?

  1. Getting Started
  2. Configuration

Profiling

Best practices when it comes to profiling

PreviousFilteringNextLineage

Last updated 3 years ago

Was this helpful?

Profiling is the process of gathering statistics on the columns. This can be challenging when working with big datasets. This page will go through the options to optimize this process.

This guide will give you full control over when profiles are being computed:

  • On-demand this is by default, introducing some latency when seeing the profiles, but you know that they are up to date.

  • Scheduled this will make sure that the profiles are computed at a convenient time (for example, off-business hours), but it can be that the profile is slightly outdated.

Below we can see that the profile has been updated since 2021-06-19 08:46:15 UTC. By default, the profile is being computed when you visit the table:

It is highly recommended to set the computation of statistics on a schedule. This will make sure that the profiles are computed outside of business hours, and that the statistics are cached in Datafold, resulting in a snappy experience by having the statistics without any delay. We can configure this on a fine-grained level in the data source configuration page:

  • Indexing is the process of looking up the metadata. This doesn't run queries against the cluster, but only looks up the table names and their corresponding columns.

  • Profiling is the actual heavy lifting. This kicks off several jobs to compute statistics on table and column level. Depending on the type it will compute the sparsity and histograms per column.

We recommend setting a crontab schedule outside of business hours, preferably on the weekend, to avoid interference with queries that are running during business hours.

After checking the Discourage manual profile refresh in the data source configuration. There will be a pop up when manually refreshing the profile:

Hope this helps when fine-tuning the profiles in Datafold. If there are any questions, please don't hesitate to reach out.

Example profiling
Configuration