Data Engineer

Core Responsibilities

• Dimensional Modeling: Expertly design and maintain enterprise-grade dimensional models (star/snowflake schemas) that translate business processes into scalable analytical structures.
• Pipeline Engineering: Build and optimize data pipelines using Spark/PySpark on Databricks and Microsoft Fabric, supporting batch, incremental, and CDC workloads with robust error handling.
• Advanced SQL & Transformation: Write production-grade SQL for complex transformations and aggregations across Lakehouse and Warehouse endpoints, focusing on performance tuning.
• Platform Mastery: Operate across the Databricks (Unity Catalog, Delta Lake) and Microsoft Fabric (OneLake, Data Factory) ecosystems, ensuring seamless architectural integration.
• Data Quality & DevOps: Implement automated testing and data quality checks within CI/CD pipelines, while maintaining clear documentation, lineage tracking, and metadata tagging.
• Efficiency & FinOps: Optimize query performance and apply FinOps principles to manage compute costs and ensure efficient resource utilization.

Requirements & Attributes

Technical Requirements

• Experience: Proven expertise in dimensional data modeling and schema design.
• Technical Stack: Hands-on experience with Spark/PySpark, Databricks, and Microsoft Fabric.
• Language Skills: Proficiency in advanced SQL and Python for data engineering.
• Operational Skills: Knowledge of CI/CD best practices, metadata management, and performance profiling.

Behavioural Attributes

• Priority Management: Ability to assess urgency, communicate trade-offs, and adapt to shifting demands.
• Analytical Problem Solving: Methodical approach to root cause analysis and implementing preventive measures.