Senior Data Engineer - Databricks - MLOps - AWS

GoTymeX is building some of world's fastest growing digital banks and the data team plays a key role in driving the bank’s vision of creating a platform that stimulates economic participation and facilitates broader financial inclusion by implementing creative best in class data and analytic solutions to achieve success in providing quality services and products to our customers and optimising the business.

As a Senior Data Engineer you will contribute to the mission by creating solutions that will directly support informed decision-making and innovation by providing clean, protected, quality and auditable data from various sources into fit for purpose data products.

In this role, you can expect to:

• Design, develop, test, deploy and monitor data pipelines in Databricks on AWS from a wide variety of data sources.
• Design, develop, test, deploy and monitor scalable code with PySpark and SQL in Databricks.
• Identify opportunities to improve internal process through code optimisation and automation.
• Build data quality dashboards, lineage flows / and or monitoring tools to utilize the data pipeline, providing active monitoring and actionable insight into overall data quality and data governance.
• Assist in migrating data from legacy systems onto newly developed solutions.
• Follow and lead best practices on all data security, retention, and privacy policies.

Requirements

• Bachelor’s degree.
• 3+ years’ experience of building ETL/ELT pipelines.
• Proven competency in solution design, development, implementation, reporting and analysis.
• Proficiency in Apache-Spark, Python and SQL languages.
• Proficiency in working with Text, Delta, Parquet, JSON, CSV, and XML data formats.
• Working knowledge of Spark structured streaming.
• AWS infrastructure experience, specifically working with S3.
• Solid understanding of git-based version control, DevOps, and CI/CD. Experience of working on Atlassian stack a plus.
• Knowledge of common web API frameworks and web services.
• Strong teamwork, relationship, and client management skills, and the ability to influence peers and senior management to accomplish team goals.
• Willingness to embrace modern technology, best practice, and ways of work.