Description

Sr. SDET (ETL, PySpark, AWS, Terraform, Data Validation)

Canada (100% Remote)

12+ Months Contract

Job Description:

Software Development Engineer in Test (SDET) with a strong foundation in data engineering, automation, and cloud technologies to join our delivery team supporting Healthfirst. The ideal candidate will play a key role in ensuring the quality, reliability, and performance of complex data pipelines and platforms in a healthcare data ecosystem.

Key Responsibilities:

  • Develop and maintain test automation frameworks for ETL/ELT pipelines, data lakes, and streaming applications.
  • Collaborate with developers, DevOps, and data engineering teams to build CI/CD pipelines integrated with automated testing.
  • Perform data validation using SQL, PySpark, and tools such as ICEDQ.
  • Contribute to infrastructure-as-code (IaC) test strategies using Terraform, GitHub Actions, or similar.
  • Implement and optimize test-driven development (TDD) approaches for Spark, Kafka, and other distributed systems.
  • Work with AWS Cloud components (e.g., S3, Glue, Lambda) and automate tests for data lake architectures.
  • Ensure compliance and data integrity for healthcare-related applications.

Required Technical Skills:

Languages & Tools:

  • Python (including Pandas and PySpark).
  • SQL (PostgreSQL, Advanced SQL constructs including Window Functions and CASE).
  • Shell scripting & Linux command line.
  • AWS: Glue, S3, Data Lakes, Cloud Practitioner Essentials.
  • Apache Spark, Apache Kafka, Apache Airflow.
  • Terraform / GitHub Actions / CI/CD.

Testing & Automation:

  • ICEDQ or similar data validation tools.
  • ETL Test Automation.
  • Test-Driven Development (TDD).
  • Structured test frameworks for data pipelines.

Platform Knowledge:

  • Data Engineering workflows and modeling.
  • Streaming and batch processing architectures.
  • Multi-threaded programming and debugging.
  • Familiarity with Delta Lake, Data Lakes on AWS.

Preferred Qualifications:

  • Bachelor’s degree in Computer Science, Engineering, or related field.
  • Healthcare or Health Insurance industry experience is a plus.
  • Hands-on experience with AWS Data Lake architectures.
  • Experience building test frameworks for distributed computing environments.
  • Certifications in AWS (e.g., Cloud Practitioner, Data Analytics, or Solutions Architect).