LLM Judge Position Consistency
~12 mincode completion
Implement position_consistency(v_ab, v_ba) returning the array [consistency, first_rate].
Where this shows up
~12 min
••••••••••••••••
8 employers weight this skill
3 AI product companies, 2 frontier labs, 1 health and bio company, 1 big tech firm, 1 autonomy company. Top match scores 93.
Python
import numpy as np
def position_consistency(v_ab, v_ba):
"""
Position-bias check for a pairwise LLM judge.
Args:
v_ab: positions picked with A shown first, shape (n,), values 0 or 1
v_ba: positions picked with B shown first, shape (n,), values 0 or 1
Returns:
array [consistency, first_rate]
"""
# YOUR CODE HERE
pass