Senior Evaluation ML Engineer

About kaiko

Delivering high quality cancer care is complex; specialists form a view of each patient's condition by reasoning across different data - CT scans, genomics context, treatment history and clinical notes.

Current AI are powerful within domains but fall short when it comes to reasoning across data or domain areas. kaiko.w, our AI assistant for oncology, aims to equip every clinician with a full understanding of their patients, helping them to reason across data as they assess each case.

We're building this in close collaboration with the Netherlands Cancer Institute (NKI) and a growing network of hospitals and research centers. We've raised significant long-term funding and have nearly doubled our team over the past year. We're now 80+ people representing 25 nationalities, based across our offices in Zurich and Amsterdam.

About the role

Kaiko's Multimodal Large Language Model (MLLM) is trained on domain-specific, high-complexity medical data. To reach clinical-grade performance, we need comprehensive, large-scale evaluation that is
clinically grounded
.

As a
Senior Evaluation
ML Engineer, you'll design and own our end-to-end evaluation stack, from gold-standard ground truths and synthetic benchmark generation to automated release-gating, with a focus on
oncology-relevant tasks and metrics
. You will partner with clinicians, external annotators and ML researchers to ensure that every signal we measure reflects real clinical decision-making and informs our model development efforts.

As a Senior Evaluation ML Engineer you will

• Build and operate
  our eval infrastructure at scale (Python + Ray/Spark, Dagster preferred) with strong CI/CD, reproducibility, and observability principles in mind.
• Source & curate
  benchmarks (public, licensed, partner-provided) and
  generate
  high-fidelity synthetic cases with controls for clinical plausibility, leakage, cohort balance, and difficulty.
• Define
  clinically meaningful task taxonomies and rubrics spanning text (clinical notes, reports), imaging (CT/MRI/PET), pathology (
  whole-slide images
  ), genomics (
  VCF
  , biomarkers), and structured
  EHR/FHIR
  data.
• Automate
  offline evaluations and
  build
  online evaluation flows (clinician-in-the-loop review, preference/ranking, A/B).
• Collaborate
  with clinicians and external partners to facilitate expert evaluations, design annotation protocols, and translate clinical questions into measurable tasks
• Maintain
  benchmark hygiene: deduplication, de-identification awareness, leakage audits, stratified sampling, etc.

You will be based in
Zurich or Amsterdam
, with the expectation of spending
~50% of your time in the office
.

About you

• Excellent
  Python
  skills and strong software engineering fundamentals (testing, modular design, CI/CD).
• Deep experience designing & operating
  evaluation or data-quality pipelines
  for ML/LLMs at scale.
• Comfortable with
  distributed
  compute (Ray, Spark), data lakehouse paradigms (Delta/Iceberg) and columnar formats (Parquet/ORC).
• Working knowledge of oncology workflows and terminology:
  staging (TNM)
  , common
  biomarkers
  , lines of therapy, response criteria (e.g.,
  RECIST
  ), typical labs and imaging follow-up.

Nice To Have

• Experience with eval frameworks (lm-eval-harness, OpenAI Evals, HF Evaluate) and preference modeling.
• Background in biomed/healthtech (bioinformatics, medical imaging, clinical decision support, translational research, real-world evidence) or graduate work in a related field.
• Safety/red-teaming for LLMs; familiarity with quality/risk practices for clinical software (e.g., MDR/SaMD concepts).
• Experience reading and operationalizing
  radiology
  ,
  pathology
  , and
  molecular
  reports for evaluation tasks.
• Hands-on experience with
  workflow orch
  estration (Dagster preferred) and monitoring/observability.
• Experience working with medical foundation models and evaluating them on benchmarks in radiology, pathology, and/or genomics
• Familiarity with medical standards/ontologies:
  FHIR/HL7
  ,
  SNOMED CT
  ,
  ICD-10/ICD-O
  ,
  LOINC
  ,
  DICOM
  ,
  VCF
  .

We are excited to gather a broad range of perspectives in our team, as we believe it will help us build better products to support a broader set of people. If you're excited about us but don't fit every single qualification, we still encourage you to apply: we've had incredible team members join us who didn't check every box.

Why kaiko

At kaiko, we believe the best ideas come from collaboration, ownership and ambition. We've built a team of international experts where your work has direct impact. Here's what we value:

• Ownership
  : You'll have the autonomy to set your own goals, make critical decisions, and see the direct impact of your work.
• Collaboration
  : You'll have to approach disagreement with curiosity, build on common ground and create solutions together.
• Ambition
  : You'll be surrounded by people who set high standards for themselves and others, who see obstacles as opportunities, and who are relentless in their work to create better outcomes for patients.

In addition, we offer

• An attractive and competitive salary, a good pension plan and 25 vacation days per year.
• Great offsites and team events to strengthen the team and celebrate successes together.
• A EUR 1000 learning and development budget to help you grow.
• Autonomy to do your work the way that works best for you, whether you have a kid or prefer early mornings.
• An annual commuting subsidy.

Our interview process

Our interview process is designed to assess mutual fit across skills, motivation, and values. It typically includes the following steps:

• Screening call: A short conversation to align on your motivation, career goals, and initial fit for the role.
• Technical interview: A deep dive into your problem-solving approach through a technical challenge, case study, or role-specific scenario.
• Onsite meeting (optional): You'll meet team members across functions to explore collaboration dynamics, team fit, and day-to-day context.
• Final executive conversation: A discussion with a member of the executive team focused on long-term alignment, cultural fit, and shared expectations for impact.