Data Engineer

  • Basis:  Full-Time
  • Closing Date:  24 Feb, 2019
  • Job Ref:  KMP-41085

Job Description

We are looking for a Data Engineer to join our growing team.

The Data Engineer will be responsible for optimising and/or re-designing our data and data pipeline architecture to support our new generation of products and data initiatives.

The right candidate has ‘hands on’ experienced with big data technologies. The hire needs to be able to leverage Azure data services to solve complex data challenges. The Data Engineer will support our teams on data initiatives to ensure consistent data delivery throughout ongoing projects.

Responsibilities for Data Engineer:

  • Create and/or maintain the data pipeline architecture.
  • Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources.
  • Design, coordinate and execute scalable data processing and analytics solutions, including technical feasibility for big data storage, processing and consumption.
  • Identify, design, and implement internal process improvements.
  • Build analytics tools that utilise the data pipeline to provide actionable insights for data scientist team members.
  • Ensure plan execution, document and share technical best practices / insights with stakeholders to assist with data-related technical issues and support their data infrastructure needs.

To Qualify for Data Engineer Role:
The candidate (you) must have 3+ years of:

  • Experience in a similar data engineering role, who has attained a graduate degree or above in Computer Science, Statistics, Informatics, Information Systems or another quantitative field.
  • Experience designing big data real-time processing solutions
  • Building and implementing big data solutions using Microsoft Data Platform and Azure Data Services; including Data Factory, Data Warehouse, Data Lake, Data Lake Analytics, Data Bricks and U-SQL
  • Working with both traditional and modern data architecture and processing concepts such as relational SQL and NoSQL databases
  • Designing data pipelines to support machine-learning solutions.
  • The right candidate must also be able to support and work with cross-functional teams in a dynamic environment in an AGILE development environment.

Technical Requirements

  • Excellent knowledge and ‘hands on’ experience with Azure cloud computing including Azure SQL DB/DW, HD Insight, Azure Data Lake Storage, Azure Data Lake Analytics, Azure Machine Learning, Stream Analytics, Azure Data Factory, Azure Data Bricks and Cosmos DB.

Desired certifications and accreditations preferably one or both or comparable:

  • Microsoft Azure Designing and implementing Big Data Analytics Solutions (70-475)
  • Microsoft Azure Designing and implementing Cloud Data Platform Solutions (70-473)
  • Experience of hybrid and/or cloud architectures that utilise Azure
  • Experience of advanced data analytics and insights techniques such as predictive, segmentation, recommendation and sentiment analytics, machine learning techniques and leveraging structured and unstructured data.

Expert knowledge and experience in some of the following;

  • Hadoop, Spark, Storm, Hive
  • NoSQL, CosmosDb, MongoDb
  • R Studio, PowerBI, Visual Studio, TFS
  • C#, R, Python, SQL, MDX, JSON, XML, .NET, C++, T-SQL, Java, JSON, Scala