We are seeking a talented and experienced Senior Data Engineer to join our growing team. The Lead Data Engineer will play a key role in designing, building, and maintaining our data infrastructure, ensuring scalability, reliability, and performance.
Who You Are:
- An engineer at heart who takes pride in a clean and powerful code base.
- Deep knowledge of Data Architecture, Data Engineering best practices and passionate about making data accessible.
- Enjoys sharing knowledge and working in a collaborative team environment.
- Ready to take ownership and make decisions.
- Data governance, security, and compliance are always top of mind.
- Previous experience in ETL/ELT Architecture: Expertise in designing and optimizing ETL/ELT pipelines to handle various data formats (CSV, JSON, Parquet, etc.) and integrating data from multiple sources (e.g., APIs, cloud storage, client Snowflake shares).
- Strong understanding of REST API principles, experience with high-volume API requests, and ability to optimize API calls and data ingestion strategies.
- Proficiency in using orchestration tools like Apache Airflow, or similar tools to automate and manage data workflows.
- Expertise in building efficient ETL/ELT workflows to enable scalable feature engineering.
- Previous experience in performance testing and optimization (data load testing/performance tuning/monitoring) for various databases, and ETL pipelines.
- Experience building and testing resilient infrastructure using IaC and cloud-specific features for disaster recovery.
- Experience working in an Agile environment.
- Experience building data products in large scale distributed systems.
- Knowledge of industry best practices and compliance standards, such as DAMA, CCPA, PIPEDA, etc.
What You Will Do:
- Work with business partners and stakeholders to understand data requirements and support data-related projects.
- Work with engineering, product teams and 3rd parties to collect required data.
- Drive data modeling and warehousing initiatives to enable and empower data consumers.
- Develop ETL/ELT pipelines to ingest, prepare and store data for the product, analysis, and data science.
- Develop and implement data quality checks, conduct QA and implement monitoring routines.
- Improve the reliability and scalability of our ETL processes.
- Develop and manage backup and recovery strategies to ensure data integrity and availability.
- Ensuring our system is architected to balance cost and latency.
- Collaborate with partners to execute compliance, security, and privacy reviews/audits.
- Deploy data infrastructure to support product, analysis, and data science.
Qualifications:
- Education: Bachelor’s degree in Computer Science, Information Technology, or a related field.
- Experience: Minimum of 6 years working with databases, preferably within a platform tool and automation environment.
- Technical Skills: Programming Languages (PySpark, Scala, Python, Snow SQL, SnowPipe, SQL, Terraform), Data Orchestration and Automation (Airflow, K8 or similar), Cloud Infrastructure and Data Management Systems (MongoDB, Snowflake, Databricks and Azure or similar).
- Problem-Solving: Strong analytical and problem-solving skills.
- Communication: Excellent verbal and written communication skills.
- Team Player: Ability to work effectively in a collaborative team environment.
- Knowledge of DevOps and mobile software develop practices and tools.
