Data Engineer

Position Summary

We are looking for an experienced Microsoft Fabric Data Engineer (3-5 years) to construct and manage our next-generation data ecosystem in Kuala Lumpur. Operating in a hybrid model, you will focus on building, scaling, and fine-tuning cloud data infrastructure using Microsoft Fabric and Databricks.

This role demands direct experience with Fabric's core environment, deep technical skill in PySpark pipeline design, expertise in analytics data modeling, and a commitment to data reliability and governance.

Core Responsibilities

1. Platform Architecture & Integration

• Design, expand, and maintain enterprise data platforms using Microsoft Fabric (OneLake, Lakehouse, Data Factory, Dataflows, Data Warehouse) and Databricks (Delta Lake, Unity Catalog, workflow orchestration).
• Unify Fabric and Databricks architectures to ensure seamless data flow, reliability, and accessibility.

2. Data Engineering & Pipeline Management

• Develop and refine efficient PySpark and Spark data pipelines to handle batch processing, real-time streaming, and Change Data Capture (CDC) workloads.
• Incorporate schema evolution strategies and automated error recovery to safeguard pipeline health.

3. Data Modeling & Query Tuning

• Establish and maintain scalable data structures using star/snowflake schemas, data vault techniques, and metadata-driven delivery models.
• Write high-performance SQL and continually tune query performance, execution plans, and storage layers across data lakes and warehouses.

4. Data Governance & Operational Quality

• Implement automated data validation, quality control frameworks, and end-to-end data lineage tracking across ecosystems.
• Enforce security protocols, access governance, and compliance standards within Unity Catalog and OneLake.

Required Skills & Experience

• Experience Level: 3 to 5 years of professional experience in data engineering roles.
• Microsoft Fabric: Demonstrated, hands-on experience working directly with Microsoft Fabric tools (OneLake, Lakehouse, Data Factory, Dataflows, Warehouse).
• Databricks Environment: Practical knowledge of Databricks, Delta Lake storage, and Unity Catalog administration.
• Technical Proficiency: Advanced command of Python, PySpark, and complex SQL.
• Data Architecture: Strong background in enterprise dimensional modeling and structured data design.
• Work Location: Based in or willing to work Hybrid in Kuala Lumpur.