Data Engineer

Data Engineer

Type: Contract (up to 12 months)

Primary Project Responsibility:
• Database platform migration:
• Migrate from Azure PostgreSQL flexible w/ Timescale Server to Azure Cosmos DB for Postgres /w Timescale.
• Platform requirements design and documentation.
• Schema design.
• Security design.
• Migration of existing schemas.
• Performance review and enhancement.
• ETL integrations.
• Parallel operations during migration.
• Process configuration:
• Review and improve existing processes; port to new system.
• Data Migration:
• Migrate up to 6TB of historical timeseries data.
• Client Migration:
• Ensure all application client services are maintained throughout the process.
• Ensure all team members and stakeholders remain engaged during the transition from one Microsoft Data platform to another.

Secondary Projects:
• Define and implement database standards for data models and SQL code.
• Assist with performance optimization of existing code base and data models.
• Recommend machine learning (ML) ecosystem and implement Proof of Concept (POC).
• Perform other database engineering tasks as needed.
• Improve data models and overall efficiencies.

Skills and Experience Desired:
• Strong analytical skills with experience in troubleshooting large-scale distributed systems.
• Expertise in data modeling and structure design.
• Ability to work independently with minimal technical supervision.
• Capability to articulate technical concepts clearly to team members and management.
• Proficiency in performance tuning, including index design, query optimization, and table partitioning.
• PostgreSQL experience: Strongly preferred.
• Timescale experience: Desired.
• Experience with time series/IoT datasets preferred.
• Azure Managed Database experience: Preferred.
• Advanced T-SQL and Python skills.
• Experience with Azure Data Factory.
• 3+ years of demonstrated technical experience in database engineering roles.

Technical Ecosystem:

You will be working with a small team in a cloud-native Azure environment. This is a fast-paced group tasked with building complex reporting and analysis tools leveraging large amounts of real-time IoT data from multiple sources.

We aim to lift and shift the database platform to a distributed environment to address scaling and performance limitations while maintaining existing functionality and ensuring minimal disruption to business application users.