Job Order #3204

Email this Job to a Friend
Posted Date: 3/7/2021
Job Category:
Position Title: Data Engineer Technical Lead
Salary Range:
Location: Markham, Ontario, Canada
Desired Skills:
Description:

JOB DESCRIPTION

Our client, a multi-national insurance company is looking for a Data Engineer Technical Lead to join their growing team of Data and Analytics experts.  They are building the next generation data platform to support the growing need for data within the organization.

 

ROLES AND RESPONSIBILITIES:

  • Participate in the architecture, design, and management of the data integration platform

  • Design, develop, and productionize ETL jobs to ingest data into Data Lake

  • Provide management and leadership of the off-shore ETL development team

  • Extract data to integrate with various business applications

  • Design and develop and maintain performance of data pipelines to ingest data into a Hadoop environment from disparate data sources (RDBMS, NoSQL, SQL, mainframe etc) using Informatica BDM

  • Parse unstructured data, semi structured data such as JSON and XML, using Informatica Data Processor

  • Analyze the Informatica PowerCenter Jobs and redesign them in BDM

  • Design and develop efficient mappings and workflows to load data into the Data Marts

  • Perform a GAP analysis between various legacy applications to migrate them to newer platforms/data marts

  • Develop queries in Hive, Impala, and PostgreSQL to extract data for analysis

  • Work with Hadoop Administrators, PostgreS DBAs to partition the hive tables, refresh metadata and various other activities, to improve ETL performance

  • Identify bottlenecks in the data pipelines/ETL jobs and optimize their performance by enhancing or redesigning

  • Write medium to complex shell scripts to pre-process source data/files

  • Identify manual processes, queries etc. and design and develop processes/ETL code to automate them

  • Participate in daily scrums; work with vendor partners, business users and the QA team throughout the development life cycle

 

SKILL REQUIRED:

  • Experience with complex data engineering related projects

  • 3+ years’ experience as a Data Engineer

  • 5+ years of experience with data integration development for data warehouses, data marts

  • 3+ years of experience working on an Hadoop platform, writing Hive or Impala queries

  • 1+ years of experience with working on AWS technologies for data pipelines, data warehouses

  • Experience managing and leading an off-shore development team

  • Strong data engineering mind-set

  • Experience with cloud platforms (ideally AWS but Azure, GCP, Snowflake)

  • Experience performance tuning of ETL jobs, Hadoop jobs, and SQL queries, using partitioning, indexing and various other techniques

  • Experience writing Shell scripts

  • Experience with Agile SCRUM methodology, ideally having used Jira, Bit bucket, GIT, Jenkins for code deployment

  • Experience with relational databases (Oracle, Teradata, PostgreSQL) and strong SQL coding skills

  • Computer Science, Engineering, Information Systems degree (or equivalent combination of skill and experience)

  • Experience with the AWS data stack (Kinesis Stream/Firehose, Lambda, Athena, Glue, RedShift, EMR) is an asset Experience as an ETL Developer, ideally with PowerCenter, is an asset

  • Experience working with Informatica BDM is an asset

  • Experience developing Spark jobs (Python or Scala) is an asset



Share this job: