PBT Group has an opportunity for a BI Big Data Engineer Specialist to interpret requirements provided by business and produce effective Big Data solutions. Programming exposure for data transformations and integrating the big data solution with existing systems. Develop information solutions from a variety of sources for both structured and unstructured data. Productionisation of machine learning models. Technical ownership of Big Data solutions for structured and unstructured data.

Duties:

• Develop and implement big data models and solutions
• Research, prototyping and design
• Feature set engineering and automation
• Design and implement ETL/ELT methodologies and technologies and the integration with big data
• Work in an Agile environment and attend to daily standups, retros and sprint planning
• Conduct root cause analysis on production issues
• Technical leadership of entire information management process of both structured and unstructured data
• Provide ongoing support and enhancement to ETL/ELT system
• Optimization of the solutions
• Implementing machine learning algorithms in production through integration
• Configuration of the Hadoop infrastructure and environment for optimal performance
• Integrate with statistical and actuarial analysts to build models
• Producing relevant technical documentation and specifications
• Estimate time and resource requirements for business requirement
• Integration of big data solutions with existing reporting and analytical solutions
• Optimization of machine learning models produced by Data Scientists.
• Develop data processing functions (DPF’s) using Java and Python

Experience:

• SQL (Advanced)
• Data Warehouse principles and practices (Advanced)
• Docker and container setup (Intermediate)
• Linux Shell Scripting (Intermediate)
• Python/R/Scala Programming (Intermediate)
• Git versioning
• Flask/Django framework
• CI/CD
• Systems Development Life Cycle (SDLC) (Intermediate)
• Spark framework Configuration (Intermediate)
• Data Security and Protection Policies (Intermediate)
• MS Excel (Intermediate)
• Kimball Methodology (Intermediate)
• ETL development using SSIS, Python & Java (Intermediate)
• Java/.net Programming (Advantageous)
• Big Data using Hadoop (Advantageous)
• Big Data Ingestion using Sqoop/Kafka (Advantageous)
• Linux administration (Advantageous)
• Distributed programming skills on a cluster environment (Advantageous)

Qualifications/ Certification:

• Matric (Essential)
• National Diploma in IT (BTech) or appropriate certification (Essential)
• Bachelor of Science (Information Systems, Computer Science, Mathematics) Advantageous