# Data Engineer (Quantexa)

**Company:** [Unison Group](https://jobs.workable.com/companies/deKpPhPMtQZga7XoPG1tSo.md)
**Location:** Singapore, Singapore
**Workplace:** on site
**Employment type:** Full-time
**Department:** Bharat

[Apply for this job](https://jobs.workable.com/view/c60e6708-0956-46b6-b7ef-acf0f148f5f5)

## Description

-   Implement data transformation, aggregation, and enrichment processes to support various data analytics and machine learning initiatives
-   Collaborate with cross-functional teams to understand data requirements and translate them into effective data engineering solutions
-   Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data
-   Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
-   Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
-   Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
-   Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
-   Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration
-   Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
-   Ensure data quality and integrity throughout the data processing lifecycle
-   Design and deploy data engineering solutions on OpenShift Container Platform (OCP) using containerization and orchestration techniques

## Requirements

-   Bachelor's or Master's degree in Computer Science, Software Engineering, or a related field
-   Must be Quantexa certified data engineer / data architect and proficient with the software.
-   Proven experience as a Data Engineer, working with Hadoop, Spark, and data processing technologies in large-scale environments
-   Proficiency in Scala programming language and familiarity with functional programming concepts
-   Experience with Quantexa tool is highly preferred.
-   In-depth understanding of Apache Spark architecture, RDDs, DataFrames, and Spark SQL
-   Strong expertise in designing and developing data infrastructure using Hadoop, Spark, and related tools (HDFS, Hive, Pig, etc)
-   Experience with containerization platforms such as OpenShift Container Platform (OCP) and container orchestration using Kubernetes
-   Proficiency in programming languages commonly used in data engineering, such as Spark, Python, Scala, or Java
-   Knowledge of DevOps practices, CI/CD pipelines, and infrastructure automation tools (e.g., Docker, Jenkins, Ansible, BitBucket)
-   Experience with Graphana, Prometheus, Splunk will be an added benefit
-   Experience integrating and working with Elasticsearch for data indexing and search applications
-   Experience with Quantexa tool is highly preferred.
-   In-depth understanding of Apache Spark architecture, RDDs, DataFrames, and Spark SQL
-   Strong expertise in designing and developing data infrastructure using Hadoop, Spark, and related tools (HDFS, Hive, Pig, etc)