IC3 - Data Engineer

Objective of the Role
Responsible for designing, developing, and maintaining robust data pipelines and architectures that support the organization’s data-driven initiatives. This role contributes to all phases of the data engineering lifecycle, ensuring scalability, reliability, and performance. Demonstrates technical leadership by solving complex problems, mentoring junior engineers, and actively improving data engineering practices.

Main Responsibilities

• Design and implement scalable and efficient data pipelines for ingesting, processing, and storing structured and semi-structured data from diverse sources.
• Collaborate with stakeholders to gather requirements, identify data sources, and define the feasibility of requested solutions.
• Develop and optimize ETL processes, ensuring the accuracy, consistency, and integrity of data throughout its lifecycle.
• Create and maintain data models, adhering to best practices for normalization and performance optimization.
• Build and manage cloud-based data architectures, including data lakes, data warehouses, and real-time streaming solutions.
• Implement monitoring and alerting systems to ensure the reliability and performance of data pipelines in production.
• Contribute to data governance initiatives by ensuring data quality, security, lineage, and compliance with relevant regulations.
• Utilize advanced data processing frameworks like Apache Spark, Apache Kafka, and Flink for batch and real-time data processing.
• Maintain and enhance CI/CD pipelines to automate data engineering workflows and ensure seamless deployment.
• Perform code reviews and enforce coding standards to ensure quality and maintainability.
• Document processes, architectures, and technical workflows to support knowledge sharing and operational continuity.
• Mentor junior engineers, sharing expertise and fostering a collaborative team environment.
• Identify opportunities for process optimization and automation to improve efficiency and reduce manual effort.
• Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to deliver impactful data solutions.
• Promote an autonomous work culture by encouraging self-management, accountability, and proactive problem-solving among team members.
• Serve as a Spin Culture Ambassador to foster and maintain a positive, inclusive, and dynamic work environment that aligns with the company's values and culture.

Required Knowledge and Experience

• Minimum 3 - 4 years of experience as a Data Engineer
• In-depth understanding of core data engineering concepts and principles, including complex ETL (Extract, Transform, Load) processes, scalable data pipelines, and advanced data warehousing techniques.
• Advanced proficiency in Python, including writing efficient and optimized code, using advanced features like decorators, generators, and context managers.
• Extensive experience with Python libraries and frameworks commonly used in data engineering, such as Pandas, NumPy, PySpark, and Dask.
• Strong knowledge of both SQL and NoSQL databases, including advanced querying, indexing, and optimization techniques.
• Experience with database design, normalization, and performance tuning.
• Advanced understanding of data modeling concepts and techniques, including star schema, snowflake schema, and dimensional modeling.
• Experience with data modeling tools and best practices.
• Proficient in various data processing methods, including batch processing, stream processing, and realtime data processing.
• Extensive experience with data processing frameworks like Apache Spark, Apache Kafka, and Apache Flink.
• Advanced knowledge of file processing concepts, including handling large datasets and working with different file formats (e.g., CSV, JSON, Parquet, Avro).
• Strong understanding of data governance principles, including data quality management, data lineage, data security, and data privacy.
• Comprehensive understanding of the end-to-end data engineering lifecycle, including data ingestion, transformation, storage, and retrieval.
• Experience with CI/CD pipelines and automation for data engineering workflows.
• Advanced understanding of various data architectures, including data lakes, data warehouses, data marts, and data mesh.
• Experience designing and implementing scalable and robust data architectures.
• Proficient with version control systems (e.g., Git) and experience managing code repositories on platforms like GitHub or GitLab.
• Advanced understanding of data visualization tools (e.g., Tableau, Power BI) and reporting techniques.
• Ability to create insightful and impactful visualizations and dashboards.
• Strong understanding of cloud computing in AWS and GCP stacks.
• Basic experience with Infrastructure as Code (IaC) tools like Terraform or CloudFormation.
• Proven experience leading projects with Objectives and Key Results (OKRs), identifying risks, and delivering significant business value.
• Ability to mentor and guide junior data engineers.
• Strong ability to communicate project status transparently, including progress, challenges, and next steps.
• Effective collaboration with cross-functional teams, including data scientists, analysts, and business