Skip to main content

This page is generated from CHANGELOG.md.

Release notes

All notable changes to this framework should be tracked here.

Developer workflow:

  • Run the script: .dev_scripts/new-branch.sh release/[your.name]/[major.minor.patch]
  • Use cheap agents since they do the good job (for example Claude Haiku 4.5, GPT-4.x...)
  • Use the prompt in GitHub: /prepare-release and ask for: I need a [major.minor.patch] release
  • Name the commit as: release [major.minor.patch]. ex: release 0.8.0
  • Below items should be only done if you manually do it.
    • Feature and fix PRs add entries only under [Unreleased].
    • Release PRs curate [Unreleased] into a concrete [X.Y.Z] section and leave a fresh [Unreleased] section at the top.
    • Keep entries short, adopter-facing, and grouped under the correct heading.

[Unreleased]

Added

Changed

Documented

[0.8.2]

Added

  • Fabric example workflows now include a personal-workspace bootstrap path for provisioning or reconciling developer-owned workspaces.
  • Fabric personal-workspace runs can attach a shared environment lakehouse while keeping developer-owned workspaces and staged content isolated.

Changed

  • Fabric deployment helpers for notebooks, pipelines, variables, and workspace sync were refactored from shell scripts into Python-based orchestration aligned with the official Fabric CI/CD flow.
  • Exact Online sync ingestion now keeps the previous checkpoint when an incremental poll returns no rows, avoiding invalid max-timestamp updates.
  • Silver custom transformations now log detected transformLogic fields and the expression applied to each field during execution.

Documented

  • Added Fabric development workflow guidance for personal workspaces, shared-environment promotion, and the split between admin and data-plane pipelines.
  • Added Fabric personal-workspace workflow documentation for local doctor and sync runs plus personal-workspace bootstrap usage.
  • Expanded the public data-plane guide with Fabric-specific guidance for carabine_files_root and explicit data_contract_location handling.

[0.8.1]

Changed

  • post_copy lifecycle hook in SourceToLanding was renamed to post_write; connector subclasses that override post_copy must rename the method to post_write.
  • Ingestion pipeline internal state and log messages updated from "copy" to "write" language throughout for consistency.

[0.8.0]

Added

  • Azure Blob Storage and SFTP ingestion connectors were added to the shared ingestion pipeline for contract-driven landing loads.
  • Azure SQL full-table and CDC connectors were added, replacing the older SQL reader split with a unified connector model.
  • Schema-drift detection and logging were added to bronze and silver processing, with example job wiring for Databricks and Fabric.
  • Databricks monitoring report assets were reorganized under the monitor workspace with deployment configuration for publishing PBIP artifacts.

Changed

  • Slowly changing dimension terminology was renamed to silver_write_strategy across contracts, tests, benchmarks, and example assets.
  • Silver deduplication now runs in the transform stage, improving duplicate handling before write strategies are applied.
  • Bronze micro-batch processing now stabilizes current-timestamp handling to avoid multiple-timestamp failures during streaming-style runs.
  • Fabric file-based ingestion can now write through BobClient or directly to a Fabric mount from the shared file connector flow.
  • The Rostarcas proxy connector was removed from the shared framework after being moved to a client-specific repository.

Documented

  • Public data-plane guidance was expanded to cover the new ingestion connector model and updated contract structure.
  • Internal implementation docs were added for schema drift handling, ODCS contract parsing, and the shared SQL connector stack.
  • Databricks report deployment guidance was added alongside refreshed monitoring report documentation.

[0.7.0]

Added

  • Contract-stage validation helpers for landing, bronze, silver, and gold contracts, with aggregated errors for invalid ODCS definitions.
  • Databricks monitoring view generation, job wiring, and refreshed monitoring semantic-model assets for the example workspace.
  • Fabric candy sample contracts and landing-to-silver job assets for an additional end-to-end example pipeline.

Changed

  • Silver pre-transform deduplication now falls back to all non-bronze columns when primary keys are absent, logging a warning instead of failing immediately.
  • Contract validation no longer requires record_modified_date_field for SCD schemas, while still validating the referenced field when it is provided.
  • Fabric monitoring report styling and ETL metric surfacing were aligned with the updated monitoring experience.

Documented

  • Expanded the public data-plane and examples guidance to cover contract validation and the new Fabric candy example.
  • Added internal implementation documentation for the shared contract validator.
  • Refreshed the documentation app UX and local deployment guidance for browsing framework capabilities and generated docs.

[0.6.0]

Added

  • Contract-driven custom transformations in the Silver pipeline, with reusable examples and focused tests.
  • Contract-driven quality checks in the Silver pipeline, with example checks for the SWAPI example datasets.
  • Fabric load-to-landing support, including reusable loader and pipeline code, ingestion connectors, and example notebooks.
  • Fabric gold-layer scaffolding and deployment examples, including contracts, notebooks, jobs, semantic models, and reports.
  • Multi-environment Fabric deployment support for separate dev and UAT rollouts.
  • A static documentation app to browse framework capabilities and generated docs locally.

Changed

  • Deduplicated SCD type 1 inputs earlier in the Silver flow to avoid merge failures on duplicate business keys.
  • Unified quarantine handling across Silver processing and example assets.
  • Refactored load-to-bronze to remove the deprecated shared loader and align Databricks autoloading with the common LandingToBronze flow.
  • Refreshed the Fabric monitoring and Star Wars report assets, including the monitoring report style.
  • Aligned the example SWAPI contracts with the updated Fabric pipeline and gold-layer structure.

Documented

  • Documented the public framework structure and workflows across architecture, data plane, infra plane, operations, examples, and tutorials.
  • Documented the shared load_to_silver implementation, including casting, SCD, deletions, quality validation, and custom transformations.
  • Documented the shared load_to_bronze pipeline and utilities, and expanded local test setup guidance for Windows.

[0.5.1]

Added

  • DELETION_PRIMARY_KEY_FIELD added for CustomProperties of SchemaObject

Changed

  • Extended data contract config for DeletionEventLogTranslator to have deletion_primary_key_field as config, to be able to map to primary key field.
  • Renamed deletion config property source_table_name to table_for_translator

Documented

[0.5.0]

Added

  • Support for deletion (4 modes): source_flag, cdc_log, deletion_event_log, snapshot_diff

Changed

  • Remove soft deletion handling from SCD type 1 and type 2
  • Remove e2e test cases for deletions
  • record_modified_date_field is now only part of SCD type 2, and it doesn't override landing_create_date anymore

Documented

[0.4.0]

Added

  • Added source-controlled release metadata in .release/manifest.yml to declare release intent, artifact type, and release-note linkage.
  • Added CHANGELOG.md as the canonical human-readable release-notes file for the framework release flow.
  • Added .azure-pipelines/scripts/release/validate_release_manifest.py to validate release metadata, enforce SemVer bump rules, derive major GUID storage paths, and emit Azure DevOps variables for downstream pipeline stages.
  • Added .azure-pipelines/azure-pipelines-framework-ci.yml to validate release metadata on pull requests and refresh the moving internal {latest-guid}/latest.zip artifact on main.
  • Added .azure-pipelines/scripts/release/manifest_utils.py as the shared manifest reader used by validation, reset, and publish-gate logic.
  • Added focused pytest coverage in .azure-pipelines/scripts/release/tests/test_release_manifest_scripts.py for manifest parsing, SemVer bump validation, and release-manifest reset behavior.

Changed

  • Split framework delivery into two tracks: CI for internal latest publishing and CD for governed release publication.
  • Refactored .azure-pipelines/azure-pipelines-framework.yml into a release-only pipeline that skips publish work when release_type is no-release.
  • Moved release artifact addressing to a derived {major-guid}/{version}.zip model instead of manually composing major-version paths inside the pipeline.
  • Moved internal latest artifact addressing to a hidden {latest-guid}/latest.zip path while keeping consumers configured through carabineFrameworkPath and carabineFrameworkVersion.
  • Changed post-release manifest reset handling to create an automated pull request instead of pushing directly to main.
  • Replaced the release publish gate's awk parsing with the shared Python manifest reader so the pipeline and scripts interpret release metadata consistently.

Documented

  • Documented the new framework CI/CD workflow in README.md, including normal development flow versus governed release flow.
  • Added regex documentation and concrete matching examples for the release validator patterns so future edits to the validator are easier to review safely.

[1.0.0]

Added

  • Initial Carabine framework baseline.
Last updated on