Data Engineer Associate

Technologies we use

Your responsibilities

• Build, orchestrate and optimize data pipelines to extract, transform, and load (ETL) data from various sources into the organization's data warehouse or data lake.
• Integrate data from diverse sources such as databases, APIs, streaming platforms, and file systems into cohesive data pipelines.
• Implement data integration solutions that support real-time, batch, and incremental data processing.
• Implement data quality checks and validation processes to ensure the accuracy, completeness, and consistency of data.
• Develop and maintain high-performance data pipelines that seamlessly integrate data from various sources into data warehouses, data lakes, and other storage solutions.
• Develop monitoring and alerting mechanisms to identify and address data quality issues proactively.
• Optimize ETL (Extract, Transform, Load) processes for efficiency, reliability, and data quality.
• Implement and manage data storage solutions, including relational databases, NoSQL databases, and distributed file systems.
• Manage the infrastructure and resources required to support data engineering workflows, including compute clusters, storage systems, and data processing frameworks.
• Implement security controls and data governance measures for an application to protect sensitive data and ensure compliance with regulatory requirements such as GDPR, CCPA, HIPAA, and PCI-DSS. Implement encryption, access controls, and auditing mechanisms to safeguard data privacy and integrity.
• Write production-ready, testable code that adheres to engineering best practices and accounts for edge cases and error handling.
• Develop comprehensive unit tests and integration tests to validate data pipeline functionality and data integrity.
• Stay up to date with the latest data engineering tools, technologies, and methodologies, and evaluate their applicability to the team's needs.
• Collaborate with business analysts, and other stakeholders to understand data requirements and translate them into robust engineering solutions.
• Work closely with other engineering teams to integrate data solutions seamlessly into the overall technology ecosystem.
• Participate actively in agile ceremonies, communicate progress, and manage dependencies effectively.

Our requirements

• A relevant Bachelors or higher-education degree in Computer Science, Data Science, or related fields.
• 1-3 years of experience in a data engineering related role, with a focus on building scalable, reliable, and high-performance data systems.
• Proficiency in programming languages such as Python, Java, or Scala.
• Extensive experience with big data technologies (e.g., Hadoop, Spark, Kafka) and cloud-based data platforms (e.g., AWS, Azure).
• Expertise in data integration and ETL tools (e.g., Talend, Informatica, Apache NiFi).
• Strong understanding of data warehousing, and data lake concepts and best practices.
• Familiarity with CI/CD pipelines and application monitoring practices.
• Certifications in data engineering (e.g., Azure Data Engineer, AWS Certified Data Engineer) are a plus.
• Proven track record of designing and implementing data pipelines, data storage solutions, and data processing workflows.
• Hands-on experience with distributed computing frameworks and cloud-based data services.
• Demonstrated ability to collaborate with cross-functional teams and communicate technical solutions effectively.
• Strong communications skills.

What we offer