Snorkel AI / Expert evaluation
At Snorkel AI, I wrote 150+ graduate-level medical and legal evaluation tasks and scoring rubrics.
Score a sample answerSnorkel AI · Expert evaluation
At Snorkel AI, I wrote tasks and rubrics for evaluating medical and legal reasoning. Try grading one fictional answer.
Drug A is approved for use alone. No evidence about A with B is supplied.
“That proves A is safe to use with B.”
The record supplies no combination evidence. The answer turns that missing evidence into a safety claim.
The record says drug A is approved alone. The AI says, “That proves A is safe to use with B.” Does the record support its conclusion?
The record only describes A alone. The answer would need evidence about A and B together to support its claim.
Approval of A alone is the supported fact. Safety of the combination is the unsupported claim. The rubric scores those separately.
The supplied record supports approval of A alone. An answer earns a positive score for stating that fact accurately and limiting it to the approved use.
No combination evidence is supplied. The answer loses marks when it converts that absence into a positive safety conclusion.
A stronger answer distinguishes the approval fact from the unresolved question and asks for evidence specific to A with B.
My documented Snorkel work involved graduate-level medical and legal evaluation tasks and rubrics. This portfolio exercise is original and illustrative, not a client task or a validated benchmark.
This exercise was written for the portfolio. Client evaluation materials are private.
Distinct roles
150+ graduate-level evaluation tasks and scoring rubrics benchmarking frontier-model reasoning in clinical decision-making and legal analysis.
Medical and Legal Expert AI Assessment Writer. Private client tasks and downstream benchmark results are not reproduced here.
Authored the reviewer standards a production legal AI product is graded against: citation accuracy, reasoning-chain validity, jurisdictional correctness, evidence quality, calibrated uncertainty, unauthorized-practice safety.
80+ expert-verified SFT gold responses and RLHF preference items a week, at a 96%+ QA pass rate.
Roles in the full record ↗