Psybergate is seeking a Data Engineer for a 12-month contract with a financial services client in Sandton. The ideal candidate will have strong Python and SQL skills, experience with Kafka, Docker, and Kubernetes, and a proven track record in building data processing pipelines, developing microservices, and designing data schemas. Responsibilities include developing and maintaining data pipelines, collaborating with teams to integrate data systems, and leveraging Kafka for real-time data streaming.
Psybergate is an IT company that builds bespoke software solutions and provides highly skilled resources to its clients. We are looking for a Data Engineer to join our financial services client based in Sandton for a 12-months contract role.
We are looking for an experienced Data Engineer with strong Python and SQL skills, as well as hands-on experience with Kafka, Docker, and Kubernetes. The ideal candidate will have a proven track record in building bespoke data processing pipelines, developing microservices, and designing data table structures and schemas. You will collaborate with cross-functional teams to integrate and optimize data systems, while working with both structured and unstructured data.Develop and maintain custom, scalable data pipelines for processing large datasets, including unstructured text and document data.Collaborate with diverse teams to understand data sources and build robust integrations with various internal and external systems.Develop microservices to handle data processing tasks, ensuring they are scalable, secure, and efficient.Design and optimize table structures, schemas, and database queries for improved data retrieval and storage performance.Leverage Kafka for real-time data streaming and processing, ensuring high availability and low-latency data delivery.Use Docker and Kubernetes for deploying and managing data applications in production environments.Work closely with data scientists, analysts, and other engineers to define requirements and deliver high-quality, reliable data solutions.Maintain data integrity and implement best practices to ensure the consistency and quality of the data being processedBachelor’s degree in Computer Science, Engineering, or a related field .Experience in developing scalable data applications, automating processes, and implementing algorithms.Hands-on experience with Kafka, including setting up and managing streams, and integrating with data pipelines. Proficiency with Docker and Kubernetes for containerization and orchestration in production environments. Experience building bespoke data processing pipelines: Expertise in handling large volumes of data from multiple sources and optimizing processing workflows.Experience designing table structures and schemas that meet both performance and business requirements. Ability to collaborate with diverse teams, understanding their needs and building integrations that allow for efficient data flow. Java experience is advantageous, particularly for implementing additional data processing logic or integration points. Experience with unstructured text and document data is a plus, especially in processing and transforming such data for analytical purposes.Please note that most of our positions are remote however candidates should be residing within the traveling distance as circumstance of the opportunity can change.
DATA ENGINEER PYTHON SQL KAFKA DOCKER KUBERNETES MICROSERVICES PIPELINES FINANCIAL SERVICES
Australia Latest News, Australia Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
Data Engineer - Asset Management SoftwareA Data Engineer is needed to support Asset Management clients using software designed for research and investment. Responsibilities include 2nd tier support, understanding client data needs, developing and maintaining data architectures, integrating systems, and preparing data for predictive modelling. The ideal candidate will have a degree in computer science, IT, mathematics, applied mathematics, actuarial science, or engineering and up to 2 years of experience.
Read more »
Data Engineer - Western Cape Century CityIT Industry News. Daily.
Read more »
Data Engineer / Business Intelligence SpecialistWe are seeking a skilled Data Engineer/Business Intelligence Specialist to join our growing team. You will play a key role in transforming data into actionable insights, supporting our enterprise data warehouse, data marts, and business intelligence reporting environments. Responsibilities include collaborating with stakeholders, data engineers, and BI analysts, analyzing business requirements, designing and implementing data solutions, and optimizing data pipelines and infrastructure.
Read more »
Data Engineer - Financial Services - 12-Month Contract in SandtonPsybergate is looking for a skilled Data Engineer to join a financial services client in Sandton, Johannesburg, South Africa. This 12-month contract role requires extensive experience with Ab Initio and a strong understanding of data engineering principles.
Read more »
Data Engineer - Join Our Team and Build Secure, Scalable Data SolutionsWe are searching for an experienced Data Engineer to join our team. This role involves designing and implementing secure, scalable, and cost-effective data solutions using on-premises technologies and cloud platforms (Azure and AWS). You will collaborate with clients and various stakeholders to understand data requirements and deliver high-quality data solutions.
Read more »
Data Engineer - Power BI & Cloud Data PlatformsWe are looking for a skilled Data Engineer to design, develop, and optimize custom Power BI reports and dashboards. This role will involve working with cloud-based data platforms and leveraging expertise in SQL, DAX, and data visualization.
Read more »