Senior Data Engineer

How you'll make an impact - Design and build production pipelines on Databricks using Spark Declarative Pipelines (SDP) and PySpark, from raw ingestion through business-ready data products. - Define all pipelines, jobs, and schedules as code in Databricks Asset Bundles, deployed to every environment through automated CI/CD. - Build data quality, monitoring, and lineage into pipelines so issues are caught and diagnosed before they reach consumers. - Turn recurring solutions into reusable frameworks, standards, and shared libraries that raise delivery speed across the team. - Own your pipelines in production — performance, cost, reliability, and incident response. - Partner with Digital Product Managers, architects, and business stakeholders to translate requirements into technical designs, and mentor engineers newer to the platform. What you'll need (Required) - Bachelor's degree in computer science, engineering, or a related technical field, plus five or more years of data engineering experience, including hands-on production experience on Databricks. - Demonstrated experience with Spark Declarative Pipelines (SDP / Delta Live Tables) — streaming tables, materialized views, expectations, Auto Loader, and CDC patterns. - Hands-on experience deploying Databricks workloads with Databricks Asset Bundles (DABs) across multiple environments. - Strong Spark and PySpark skills, including performance tuning, and production-quality Python beyond notebook scripting. - Working knowledge of Delta Lake, Unity Catalog, medallion architecture, and strong analytical SQL. - Experience with Git-based CI/CD in a shared repository — code review, automated validation, and promotion across environments. - Demonstrated ability to take ambiguous requirements through design to production independently, and to make and defend sound technical decisions. - "Experience with interoperable catalog architectures across Unity Catalog and Snowflake Horizon, including Iceberg REST Catalog and catalog-linked databases for cross-platform table access without data duplication." -  "Working knowledge of Apache Iceberg as a table format, including managed versus external Iceberg tables and the performance trade-offs of cross-engine reads." What else we look for (Preferred) - Hands-on experience with Spark Declarative Pipelines (SDP / Delta Live Tables), including streaming tables, materialized views, expectations, Auto Loader, and CDC patterns. - Hands-on experience deploying Databricks workloads with Databricks Asset Bundles (DABs) across multiple environments. - Strong Spark and PySpark development skills, including performance tuning, and production-quality Python beyond notebook scripting. - Working knowledge of Delta Lake, Unity Catalog, medallion architecture, and strong analytical SQL. - Experience with Git-based CI/CD for data platforms, including code review, automated validation, and promotion across environments. - Experience with cloud data platforms on AWS, metadata-driven ingestion frameworks, and infrastructure-as-code. - Experience with interoperable catalog architectures across Unity Catalog and Snowflake Horizon, including Iceberg REST Catalog and catalog-linked databases for cross-platform table access without data duplication. - Working knowledge of Apache Iceberg as a table format, including managed versus external Iceberg tables and the performance trade-offs of cross-engine reads. - Familiarity with the broader modern data ecosystem, such as Snowflake, dbt, Kafka, and Airflow. - Experience integrating enterprise and clinical source systems, such as Epic, SAP, or Salesforce, or migrating workloads from legacy ETL platforms onto a lakehouse. - Practical understanding of governance, quality, security, validation, and support expectations in a regulated enterprise environment. - Ability to provide technical guidance, coach team members, and contribute to reusable standards, documentation, and delivery practices.