Data platform / modernization
From fragmented Azure resources to a unified Fabric platform
A full rebuild consolidated years of Azure sprawl into a governed Microsoft Fabric platform with lower cost, faster processing, and hourly operational visibility.
60%+
lower infrastructure cost
20 -> 3m
overnight batch runtime
Hourly
operational refresh
300+
users and leaders supported
System context
Environment
Analytics for a manufacturing, reverse-logistics, and sales organization supported more than 300 team members across operations, finance, HR, sales, and executive leadership.
Constraint
Years of changing requirements and ownership had left Data Factory, Azure SQL, Blob Storage, Functions, Logic Apps, Power BI models, and reports fragmented and difficult to manage. Cross-department analysis was slow, refreshes lagged, and development happened directly in one production workspace.
Architecture
S3 + SFTP + business systems
Fabric ingestion
Bronze + silver Lakehouses
dbt + SQL gold models
Semantic models
Reports + agents
Delivery notes
Rebuild, do not just port
The migration moved through discovery, consolidation, and rebuild. Rather than reproducing legacy Data Factory jobs, Paul rewrote processing logic as scalable PySpark notebooks with parallel execution and a medallion architecture.
Native S3 shortcuts, SFTP labor feeds from Paycom, Business Central integrations, and internal sources converged in Fabric Lakehouses. dbt and SQL produced governed gold models for shared semantic and reporting layers.
Establish an operating model
The rebuilt platform introduced Git-backed source control, monitoring, separate development and production environments, and controlled deployment pipelines in place of a single shared workspace.
Key Vault protected secrets, service principals authenticated resource access, workspace roles governed Fabric, and Power BI row-level security separated financial information from broader operating audiences.
Move from daily to operational
Overnight batch processing fell from roughly 20 minutes to 3 minutes. A parallel hourly path refreshed the same S3 operational data throughout the day, giving leaders current visibility without sacrificing the governed batch layer.
The completed migration retired obsolete Azure resources, reduced monthly infrastructure cost by more than 60%, and maintained near-continuous availability across reporting and infrastructure.
Add an intelligent access layer
Production Microsoft Foundry agents connected to governed Fabric data through Data Agents, Fabric MCP, and custom tools. The platform could answer plain-language questions while retaining the semantic, identity, and access controls established underneath it.
Paul's ownership
Paul identified the backend as the reporting constraint and independently led discovery, architecture, stakeholder alignment, implementation, migration, modeling, Power BI delivery, governance, deployment, training, and ongoing operation throughout the engagement.
Capabilities
- + Platform architecture
- + Data engineering
- + Migration
- + Governance
- + BI delivery
- + FinOps
Selected stack
- + Microsoft Fabric
- + PySpark
- + dbt
- + SQL
- + Power BI
- + Azure Key Vault
- + Git
- + S3
Supporting evidence
Curated / anonymized
Sanitized PySpark and SQL design extracts
Inspect the patterns behind this system.
Six implementation extracts map the platform's core data path: versioned intake, Bronze lineage, Silver snapshot and upsert processing, freshness controls, and Gold reporting models.
Repository coverage
- + Incremental file ingestion and lineage
- + Snapshot and keyed Delta processing
- + Timezone-aware freshness checks
- + Asset-level Gold reporting models
- + Calendar-complete operating summaries
Evidence boundary
The folder demonstrates engineering patterns rather than a released Fabric workspace. Connectors, dbt metadata, semantic models, deployment configuration, access controls, run logs, billing, and adoption evidence are not included.
Next system / 03
Turning analyst requests into governed, self-service answersA useful first conversation