Senior Big Data Engineer

Responsibilities

• Technical Leadership: Provide technical leadership and guidance to junior data engineers, mentoring them on best practices, design principles, and emerging technologies in the field of data engineering.
• Data Architecture: Design and implement scalable, robust, and efficient data architectures, including data lakes, data warehouses, and data pipelines, to support the organization's data requirements.
• Data Pipeline Development: Develop and maintain ETL (Extract, Transform, Load) processes and data pipelines to ingest, process, and transform large volumes of data from various sources, ensuring data quality and reliability.
• Data Modeling: Design and implement data models and schemas to organize and structure data for efficient storage, retrieval, and analysis, considering business requirements and data governance principles.
• Data Integration: Integrate data from diverse sources, such as databases, APIs, files, and streaming platforms, ensuring seamless data flow and interoperability across systems.
• Data Transformation and Enrichment: Implement data transformation and enrichment processes to derive insights, perform aggregations, and create derived datasets for analytics, reporting, and machine learning.
• Data Governance and Compliance: Establish and enforce data governance policies, standards, and best practices to ensure data integrity, security, privacy, and compliance with regulatory requirements.
• Performance Optimization: Optimize data pipelines and queries for performance, scalability, and efficiency, leveraging techniques such as partitioning, indexing, and query optimization.
• Collaboration: Collaborate with cross-functional teams, including data scientists, analysts, software engineers, and business stakeholders, to understand data requirements and deliver data solutions that meet business needs.

Required Experience

• 7+ Years of overall software development experience in enterprise systems.
• 6+ years of Data Management experience in an enterprise environment.
• 6+ years of experience with SQL.
• 5+ years of experience with AWS S3, Spark and Glue using PySpark (Scala nice to have).
• 5+ years working with (Snowflake, Databricks)
• Strong knowledge of metadata management, data governance, and data lineage.
• In-depth knowledge of Agile Software Development practices and how to leverage these to streamline project execution.
• Possess the ability to decompose business problems into smaller parts to correctly solution, taking into consideration the impact on other subsystems.
• Team player with excellent problem-solving and troubleshooting skills.
• Familiar with software engineering practices, including CI/CD pipelines, Automated Testing, and Performance Engineering.
• Working knowledge of SDLC, especially regarding improving agile delivery metrics.
• Strong communication and team/technical leadership skills.
• Bachelor’s degree in Computer Science, MIS, or a related field.

Preferred Experience

• AWS certification preferred
• Financial services experience
• Consulting experience.