Data Engineering Architect

Why Join Us?

We are a leading organization dedicated to driving digital transformation through innovative data solutions. Our team plays a critical role in managing data ingestion, access to curated datasets, and distribution of data to users and tools across various business functions. We support capital allocation, business development, investment evaluation, and franchise-building efforts, ensuring that our data solutions align with strategic objectives.

As part of our commitment to digital excellence, we focus on leveraging cloud technologies, modern data architectures, and advanced analytics to create scalable and efficient data solutions that drive business impact.

Position Summary:

We are seeking a Data Engineering Architect to design and implement data solutions built on the Microsoft Azure and Databricks technology stack. This role involves architecting and implementing end-to-end data engineering solutions, focusing on ELT processes, data enrichment, analytical workloads, and controlled data access.

The ideal candidate will have strong hands-on experience with Databricks, Azure Data Services, and large-scale data processing while also providing strategic guidance to business stakeholders. This role requires both an individual contributor mindset and the ability to work collaboratively with other engineers and outsourced teams.

Key Responsibilities:

o   Design and document data engineering solutions leveraging Azure and Databricks.

o   Develop, implement, and optimize ELT/ETL pipelines for structured and semi- structured datasets.

o   Implement solutions ensuring data lineage, governance, and security

compliance.

o   Optimize Databricks workloads, managing compute resources efficiently.

o   Work with internal and outsourced data engineers to coordinate daily engineering activities.

o   Communicate solution designs and updates effectively to business and technical stakeholders.

o   Lead release management and deployment processes for data solutions.

o   Debug and enhance existing data pipelines and architectures.

o   Implement CI/CD best practices in data engineering using Azure DevOps or equivalent tools.

o   Ingest and process data from relational databases and secure REST APIs.

o   Utilize data visualization tools such as Power BI or Spotfire for reporting and insights.

Required Qualifications & Skills:

o   Bachelor’s/Master’s/PhD in Computer Science, Data Engineering, or a related field.

o   10+ years of experience in data engineering, cloud data platforms, and architecture.

o   Strong Databricks experience, including Medallion Architecture.

o   Proficiency in Python, R, Spark, SQL, and experience integrating Python notebooks into data pipelines.

o   Experience with Azure Data Factory, Azure Synapse, and Data Lake Storage.

o   Hands-on experience managing large-scale data ingestion, transformation, and analytics pipelines.

o   Experience with Azure cloud-based data solutions.

o   Familiarity with CI/CD, Infrastructure as Code, and automation frameworks.

o   Understanding of data governance, lineage, and security best practices.

o   Strong communication skills with the ability to translate complex technical concepts into business-friendly language.

o   Experience working in cross-functional teams, including business analysts, data scientists, and IT teams.

o   Self-starter attitude with problem-solving skills to drive innovative solutions.