Author Image

CEO & Co-founder of Visivo

Developer-First BI Workflows with Automation and Control

Explore how developer-first BI workflows bring automation, control, and predictability to analytics through code-driven practices.

Developer-first BI workflow diagram

Treat BI development like software development and most of the chaos goes away. A developer-first BI workflow puts dashboards, charts and metric definitions in code, runs them through automation, and replaces manual clicking with a process a team can review and repeat. The gain is not only speed. It changes what a small team can safely ship.

What a developer-first BI workflow is

A developer-first BI workflow treats analytics as code. Version control, testing, automation and deployment pipelines replace manual GUI work. The DORA State of DevOps Report found that elite performers have significantly lower change failure rates than low performers. The same gap exists between dashboards crafted by hand and dashboards engineered like software.

In a traditional BI workflow, analysts work in isolation, click through interfaces, deploy changes by hand, and hope nothing breaks. According to Gartner analyst Nick Heudecker, 85% of big data projects fail. Developer-first workflows flip this model:

  • Manual clicks become code. Dashboards, charts, and metric definitions live in text files.
  • Isolated work becomes reviewed work. Every change is a pull request that a teammate reads before it ships.
  • Hope becomes tests. Assertions on the data run before a change reaches production.

The role of automation

Automation removes the repetitive steps and makes every run the same.

Scheduled pipeline runs. An orchestrator runs the whole chain on a schedule, from extract through dashboard tests:

# airflow DAG for automated BI pipeline
from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'analytics-team',
    'retries': 2,
    'retry_delay': timedelta(minutes=5)
}

with DAG(
    'bi_pipeline',
    default_args=default_args,
    schedule='0 */2 * * *',  # Every 2 hours
    catchup=False
) as dag:

    extract_data = BashOperator(
        task_id='extract_data',
        bash_command='python extract_sources.py'
    )

    transform_data = BashOperator(
        task_id='transform_data',
        bash_command='dbt run --models staging marts'
    )

    validate_data = BashOperator(
        task_id='validate_data',
        bash_command='dbt test'
    )

    update_dashboards = BashOperator(
        task_id='update_dashboards',
        bash_command='visivo run'
    )

    test_dashboards = BashOperator(
        task_id='test_dashboards',
        bash_command='visivo test'
    )

    extract_data >> transform_data >> validate_data >> update_dashboards >> test_dashboards

Tests on dashboard data. Write assertions on the data behind each chart. visivo test runs them, so a CI job fails before a wrong number ships:

# project.visivo.yml
name: sales_analytics

models:
  - name: orders
    sql: SELECT order_date, amount FROM orders

insights:
  - name: revenue-by-month
    props:
      type: bar
      x: ?{ date_trunc('month', ${ref(orders).order_date}) }
      y: ?{ sum(${ref(orders).amount}) }

tests:
  - name: revenue-is-never-negative
    assertions:
      - ">{ numpy.min( ${ref(revenue-by-month).props.y} ) >= 0 }"
  - name: every-month-has-a-value
    assertions:
      - ">{ len( ${ref(revenue-by-month).props.x} ) == len( ${ref(revenue-by-month).props.y} ) }"

Deployment through CI/CD. One pipeline compiles, tests and deploys. Production waits for a manual approval:

# .gitlab-ci.yml
stages:
  - validate
  - test
  - deploy

validate:
  script:
    - visivo compile
    - sqlfluff lint queries/

test:
  script:
    - visivo run
    - visivo test

deploy:staging:
  script:
    - visivo run
    - visivo deploy -s staging

deploy:production:
  script:
    - visivo run
    - visivo deploy -s production
  when: manual
  only:
    - main

Developer control through code

When the configuration is code, the developer controls every layer of the BI stack.

The whole project in one file. A Visivo project declares its source, models, Insights, charts and dashboards in YAML:

# project.visivo.yml
name: sales_performance

sources:
  - name: warehouse
    type: postgresql
    host: ${env.PG_HOST}
    database: analytics
    username: ${env.PG_USER}
    password: ${env.PG_PASSWORD}

models:
  - name: daily_sales
    source: ${ref(warehouse)}
    sql: SELECT sale_date, region, amount FROM daily_sales_summary

insights:
  - name: sales-by-region
    props:
      type: bar
      x: ?{ ${ref(daily_sales).region} }
      y: ?{ sum(${ref(daily_sales).amount}) }

charts:
  - name: sales-chart
    insights:
      - ${ref(sales-by-region)}

dashboards:
  - name: Sales Performance
    rows:
      - height: medium
        items:
          - chart: ${ref(sales-chart)}

Credentials stay out of the file. Each ${env.VAR_NAME} reads from the environment or a .env file, so the same project runs on a laptop and in CI.

Predictable deployment steps. Wrap deployment in a script that runs the same sequence every time, and rollback becomes the same script pointed at the previous version:

# deployment.py
class DashboardDeployment:
    def __init__(self, environment):
        self.env = environment
        self.config = load_config(environment)

    def deploy(self, dashboard_name, version):
        # Always follow the same pattern
        self.validate(dashboard_name)
        self.run_tests(dashboard_name)
        self.backup_current()
        self.deploy_new_version(version)
        self.verify_deployment()
        self.update_documentation()

    def rollback(self):
        # Predictable rollback process
        previous = self.get_previous_version()
        self.deploy(previous, immediate=True)

Infrastructure as code. The database and cache behind the dashboards are declared the same way:

# infrastructure/bi-resources.tf
resource "aws_db_instance" "bi_database" {
  identifier = "bi-analytics-db"
  engine     = "postgres"
  engine_version = "13.7"
  instance_class = "db.r5.xlarge"

  allocated_storage = 100
  storage_encrypted = true

  backup_retention_period = 30
  backup_window = "03:00-04:00"
}

resource "aws_elasticache_cluster" "bi_cache" {
  cluster_id = "bi-dashboard-cache"
  engine     = "redis"
  node_type  = "cache.r6g.large"

  num_cache_nodes = 2
  port = 6379
}

Transitioning to developer-first workflows

The move is part culture and part tooling.

Export what already exists. Start by pulling GUI-built dashboards out as files. The script below sketches the shape; the keys you write depend on the tool you migrate to:

# migration script
def migrate_dashboard_to_code(dashboard_id):
    """Convert GUI-created dashboard to code"""

    # Extract current configuration
    config = bi_tool.export_dashboard(dashboard_id)

    # Convert to YAML
    yaml_config = {
        'version': '1.0.0',
        'name': config['name'],
        'data_source': config['connection'],
        'charts': convert_charts_to_yaml(config['charts']),
        'filters': convert_filters_to_yaml(config['filters'])
    }

    # Write to file
    with open(f'dashboards/{dashboard_id}.yml', 'w') as f:
        yaml.dump(yaml_config, f)

    # Add to version control
    git_add_commit(f'dashboards/{dashboard_id}.yml',
                   f'Migrate {config["name"]} to code')

Put everything in Git. One repository holds the dashboards, the shared queries, the tests and the deployment scripts:

# BI repository structure
bi-analytics/
├── .github/
│   └── workflows/
│       └── ci-cd.yml
├── dashboards/
│   ├── executive/
│   ├── operational/
│   └── analytical/
├── queries/
│   └── shared/
├── tests/
│   ├── unit/
│   └── integration/
├── docs/
└── scripts/
    └── deployment/

Require review and green tests. A merge rule blocks any change that lacks approvals or a passing test run:

# .mergify.yml
pull_request_rules:
  - name: automatic merge for approved PRs
    conditions:
      - "#approved-reviews-by>=2"
      - status-success=continuous-integration
      - status-success=dashboard-tests
      - status-success=performance-tests
    actions:
      merge:
        method: squash

Adopt in phases. Few teams do all of this at once. A workable order:

  1. Phase 1: Version control configurations
  2. Phase 2: Add automated testing
  3. Phase 3: Implement CI/CD pipelines
  4. Phase 4: Full infrastructure as code
  5. Phase 5: Self-service analytics platform

The benefits compound at each phase:

# Maturity progression
phase_1_benefits:
  - Change history tracking
  - Basic collaboration

phase_2_benefits:
  - Catch errors before production
  - Increased confidence

phase_3_benefits:
  - Automated deployments
  - Rapid iteration

phase_4_benefits:
  - Complete reproducibility
  - Disaster recovery capability

phase_5_benefits:
  - Self-service analytics
  - Infinite scalability

Developer-first BI is not about making analytics harder. It is about making it reviewable, testable and repeatable, with the same practices your engineering team already runs. Visivo is built for this workflow: the project is YAML in your repository, visivo test runs the assertions, and visivo deploy -s staging ships what passed. The get started guide walks through the setup, and the examples gallery shows what the finished projects look like.

Install command copied