Senior Azure Data Engineer - Mexico
Job Summary As a Senior Azure Data Engineer, you play a crucial role in leveraging cutting-edge Azure data technologies to drive data transformation, analytics, and machine learning initiatives. We seek a skilled professional with exceptional technical expertise and a passion for delivering data-driven solutions.
You will work with large, complex datasets and collaborate with cross-functional teams to design, build, and optimize data pipelines, machine learning workflows, and cloud data platform integrations. Your focus will include transforming raw data into actionable insights, ensuring high-performance distributed computing, and staying at the forefront of advancements in Azure and big data technologies. Your efforts will enable the organization to maximize the value of its data assets by implementing scalable and efficient cloud-based solutions.
Main Responsibilities
Data Engineering & Transformation
Work with large, complex datasets and design scalable data pipelines on Azure Databricks using PySpark and Spark Pools.
Transform raw data into structured, actionable insights for data science and analytics use cases.
Machine Learning Development
Build, deploy, and maintain machine learning models in Azure Databricks using PySpark and frameworks like MLlib or TensorFlow.
Implement end-to-end machine learning workflows, from data collection to model deployment.
Cloud Data Platform Integration
Design and optimize solutions for integrating data from various sources such as Azure Blob Storage, Azure Data Lake, and SQL/NoSQL databases.
Leverage Databricks Notebooks and Delta Lake for advanced data processing and analysis.
Performance Optimization
Execute large-scale data processing jobs efficiently using Spark Pools.
Fine-tune Spark configurations and cluster resources to optimize distributed data processing tasks.
Job Requirements
Education
Bachelor's or Master's degree in Computer Science, Data Science, Mathematics, Statistics, or a related field.
Experience
7+ years of experience in data engineering, focusing on big data technologies and Azure cloud platforms.
3+ years of experience developing data integrations using in Azure Databricks, PySpark, Spark Pools, and large-scale data processing.
Technical Skills
Proficient in Python (specifically PySpark) and data analysis libraries like Pandas, NumPy, and SciPy.
Experience with Spark SQL, DataFrames, and RDDs for data processing and transformation.
Hands-on experience with Azure Data Lake, Azure Blob Storage, and Azure Synapse Analytics.
Familiarity with Databricks Notebooks, Delta Lake, and MLflow for model tracking and management.
Expertise in creating, optimizing, and managing Spark Pools for distributed computing.
Machine Learning
Knowledge of machine learning algorithms, model training, evaluation, and deployment.
Experience with tools like MLlib, TensorFlow, Keras, or scikit-learn for model development.
Soft Skills
Strong problem-solving abilities and analytical thinking.
Excellent communication skills to collaborate effectively with technical and non-technical stakeholders.
Ability to thrive in a fast-paced, agile environment while managing multiple tasks.
Preferred Skills (Nice-to-Have)
Experience with Apache Kafka or Azure Event Hubs for real-time data streaming.
Familiarity with CI/CD pipelines and version control tools (e.g., Git, Azure DevOps).
Knowledge of Docker and containerization for machine learning model deployment.
Experience with additional Azure tools like Azure Machine Learning Studio, Azure Functions, or Power BI for analytics.
Familiarity with Tableau or other data visualization tools to create interactive dashboards.
Job Type: Contract
Pay: $17.00 - $25.00 per hour
Expected hours: 40 per week
Schedule:
8 hour shift
Work Location: Remote
Job Summary As a Senior Azure Data Engineer, you play a crucial role in leveraging cutting-edge Azure data technologies to drive data transformation, analytics, and machine learning initiatives. We seek a skilled professional with exceptional technical expertise and a passion for delivering data-driven solutions.
You will work with large, complex datasets and collaborate with cross-functional teams to design, build, and optimize data pipelines, machine learning workflows, and cloud data platform integrations. Your focus will include transforming raw data into actionable insights, ensuring high-performance distributed computing, and staying at the forefront of advancements in Azure and big data technologies. Your efforts will enable the organization to maximize the value of its data assets by implementing scalable and efficient cloud-based solutions.
Main Responsibilities
Data Engineering & Transformation
Work with large, complex datasets and design scalable data pipelines on Azure Databricks using PySpark and Spark Pools.
Transform raw data into structured, actionable insights for data science and analytics use cases.
Machine Learning Development
Build, deploy, and maintain machine learning models in Azure Databricks using PySpark and frameworks like MLlib or TensorFlow.
Implement end-to-end machine learning workflows, from data collection to model deployment.
Cloud Data Platform Integration
Design and optimize solutions for integrating data from various sources such as Azure Blob Storage, Azure Data Lake, and SQL/NoSQL databases.
Leverage Databricks Notebooks and Delta Lake for advanced data processing and analysis.
Performance Optimization
Execute large-scale data processing jobs efficiently using Spark Pools.
Fine-tune Spark configurations and cluster resources to optimize distributed data processing tasks.
Job Requirements
Education
Bachelor's or Master's degree in Computer Science, Data Science, Mathematics, Statistics, or a related field.
Experience
7+ years of experience in data engineering, focusing on big data technologies and Azure cloud platforms.
3+ years of experience developing data integrations using in Azure Databricks, PySpark, Spark Pools, and large-scale data processing.
Technical Skills
Proficient in Python (specifically PySpark) and data analysis libraries like Pandas, NumPy, and SciPy.
Experience with Spark SQL, DataFrames, and RDDs for data processing and transformation.
Hands-on experience with Azure Data Lake, Azure Blob Storage, and Azure Synapse Analytics.
Familiarity with Databricks Notebooks, Delta Lake, and MLflow for model tracking and management.
Expertise in creating, optimizing, and managing Spark Pools for distributed computing.
Machine Learning
Knowledge of machine learning algorithms, model training, evaluation, and deployment.
Experience with tools like MLlib, TensorFlow, Keras, or scikit-learn for model development.
Soft Skills
Strong problem-solving abilities and analytical thinking.
Excellent communication skills to collaborate effectively with technical and non-technical stakeholders.
Ability to thrive in a fast-paced, agile environment while managing multiple tasks.
Preferred Skills (Nice-to-Have)
Experience with Apache Kafka or Azure Event Hubs for real-time data streaming.
Familiarity with CI/CD pipelines and version control tools (e.g., Git, Azure DevOps).
Knowledge of Docker and containerization for machine learning model deployment.
Experience with additional Azure tools like Azure Machine Learning Studio, Azure Functions, or Power BI for analytics.
Familiarity with Tableau or other data visualization tools to create interactive dashboards.
Job Type: Contract
Pay: $17.00 - $25.00 per hour
Expected hours: 40 per week
Schedule:
8 hour shift
Work Location: Remote
