Data Engineer

The Data Engineer is responsible for designing, building, and maintaining the group’s data infrastructure to support application development, data warehousing, and reporting needs. The role requires handling data from multiple sources, ensuring it is properly stored, processed, and made accessible to the teams that need it, including developers, analysts, and business stakeholders.

 

The Data Engineer will collaborate with application development teams to integrate databases into software platforms, set up and maintain data pipelines for analytics and BI tools, and ensure data is clean, secure, and readily available for reporting and insights generation. The ideal candidate will have a strong understanding of both traditional databases and modern data processing frameworks, with the ability to manage data across cloud and on-premises environments.

 

Key Responsibilities

 

1. Database Design, Management, & Optimization

• Design, develop, and maintain the databases (both SQL and NoSQL) that support the group’s applications and platforms.
• Collaborate with developers to define database architecture that supports business logic and application requirements.
• Ensure the databases are optimized for performance, availability, and scalability, handling large volumes of data in a telecom environment.
• Implement data models and structures to support different business needs, from operational data to analytical data.

 

2. Data Pipelines & ETL (Extract, Transform, Load)

• Build and maintain efficient, scalable, and reliable ETL pipelines to ingest, clean, and transform data from various internal and external sources into usable formats for reporting, analytics, and BI.
• Automate data pipelines to handle real-time and batch data processing as needed for different use cases (e.g., customer data, billing, traffic analysis).
• Collaborate with the development and operations teams to ensure data flow between systems is seamless, secure, and monitored effectively.

 

3. Data Warehousing

• Design and implement data warehouses to store large volumes of structured and unstructured data, providing a single source of truth for the group’s business and technical data.
• Manage the extraction and aggregation of data into the data warehouse from various systems, including telecom platforms, customer databases, and operational systems.
• Ensure the warehouse is optimized for performance and supports complex queries, reporting, and analytics for business users and analysts.

 

4. Reporting & Business Intelligence (BI)

• Collaborate with the BI team and business analysts to ensure data is structured and organized in a way that supports efficient reporting and insight generation.
• Support the development of BI dashboards, reports, and analytics solutions by ensuring data availability and integrity in platforms such as Power BI, Tableau, or similar.
• Work closely with business stakeholders to understand reporting needs and develop data structures that support dynamic and accurate reporting.

 

5. Data Governance, Security, & Compliance

• Implement data governance policies to ensure that data is accurate, secure, and consistent across all systems.
• Ensure proper access controls and security measures are in place to protect sensitive data (e.g., customer data, financial data) from unauthorized access.
• Ensure compliance with telecom industry regulations, data privacy laws (such as GDPR), and any other relevant data protection standards.
• Manage data quality initiatives by identifying and resolving data integrity issues across databases and platforms.

 

6. Performance Monitoring & Optimization

• Monitor data systems for performance, ensuring that databases, ETL pipelines, and data warehouses run efficiently with minimal downtime.
• Troubleshoot data pipeline issues and implement fixes to ensure the smooth flow of data between systems.
• Continuously identify opportunities for performance improvements, including database tuning, query optimization, and data processing improvements.

 

7. Collaboration & Continuous Improvement

• Work closely with developers, DevOps, and application teams to ensure seamless integration of data into applications and platforms.
• Collaborate with analysts, data scientists, and business stakeholders to provide data support for analytics and BI initiatives.
• Stay up to date with the latest trends, technologies, and best practices in data engineering, suggesting improvements and new approaches to data architecture and pipelines.

 

Qualifications and Experience

 

Required:

• Bachelor’s Degree in Computer Science, Information Technology, Data Science, or a related field.
• 7+ years of experience in data engineering, database management, or a similar role, preferably in the telecom or VAS industry.
• Proficiency in SQL (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).
• Hands-on experience with ETL tools (e.g., Apache NiFi, Talend, AWS Glue) and building data pipelines.
• Experience with data warehousing platforms such as Amazon Redshift, Google BigQuery, or Snowflake.
• Familiarity with BI tools like Tableau, Power BI, Looker, or similar platforms.
• Strong understanding of cloud data services, including AWS, Azure, or Google Cloud (e.g., S3, RDS, BigQuery).
• Knowledge of data security best practices and regulations, such as ISO271001 GDPR, POPIA and data encryption.
• Experience with programming languages such as Python, Java, or Scala for data processing and scripting.

 

Preferred:

• Experience in telecom VAS platforms and understanding the data challenges unique to telecom environments (e.g., call data records, SMS, USSD, billing data).
• Experience with big data technologies such as Apache Hadoop, Apache Spark, or Kafka.
• Familiarity with real-time data streaming and processing.
• Knowledge of machine learning pipelines or data processing for AI/ML purposes.

 

Skills & Competencies

• Data Modelling: Strong skills in data modelling for both relational and non-relational databases, ensuring optimal data structures for operational and analytical needs.
• Problem-Solving: Analytical mindset with the ability to troubleshoot and resolve complex data issues, ensuring the smooth operation of data pipelines and storage systems.
• Collaboration: Ability to work effectively with cross-functional teams, including software developers, BI teams, and business analysts.
• Communication: Excellent communication skills to explain complex data concepts to both technical and non-technical stakeholders.
• Attention to Detail: Strong attention to detail to ensure data accuracy, integrity, and security.
• Continuous Learning: Ability to stay up to date with the latest data engineering tools, technologies, and best practices to drive continuous improvement.

 

Key Metrics for Success

• Reliable, scalable, and optimized data pipelines that support the group’s VAS applications and business intelligence needs.
• High data availability, performance, and accuracy across all platforms, minimizing downtime and data issues.
• Positive feedback from business stakeholders on data availability, quality, and the efficiency of reporting and BI tools.
• Effective implementation of security and compliance measures for data management, ensuring data protection and legal compliance.
• Continuous optimization of database and data processing performance, enabling the group to scale operations as needed.

 

Why Join Us?

• Be a critical part of building the data infrastructure for a growing digital solutions group, contributing to impactful applications and business insights.
• Work with cutting-edge technologies in data engineering, from cloud platforms to modern data processing frameworks.
• Collaborate with talented teams to deliver data-driven solutions that enhance business performance and customer experience.
• Grow professionally in a dynamic, fast-paced, and technology-driven environment, with opportunities for career advancement.