docs(experiments): evaluation value cannot be None

[hassiebp]

Important

Removes None as a valid type for value in Evaluation class in experiment.py, updating the __init__ method and class docstring accordingly.

• Behavior:
  • Removes None as a valid type for value in Evaluation class in experiment.py.
  • Updates __init__ method to enforce value as Union[int, float, str, bool].
  • Updates class docstring to reflect removal of None as a valid value type.

^{This description was created by for 306529c. You can customize this summary. It will automatically update as commits are pushed.}

Disclaimer: Experimental PR review

Greptile Overview

Greptile Summary

This PR removes None as a valid value for Evaluation.value by updating the type hint from Union[int, float, str, bool, None] to Union[int, float, str, bool] and removing the corresponding documentation line.

Issue Found:
The PR incompletely updates the documentation. While the type hint and the main documentation list are updated, two code examples in the docstring (lines 127 and 173) still show value=None being used in error handling scenarios. These examples now contradict the new type constraint and will cause type errors if users follow them.

Impact:

• Users following the examples will encounter type errors
• The documentation is inconsistent, which may confuse developers
• The examples need to be updated to show alternative error handling patterns (e.g., returning None from the evaluator function to skip the evaluation, or using sentinel values)

Confidence Score: 2/5

• This PR has critical documentation inconsistencies that will mislead users
• The type hint change is correct, but the documentation examples contradict it. This creates a breaking inconsistency where examples demonstrate invalid usage that will fail at runtime or with type checkers.
• langfuse/experiment.py requires attention - the docstring examples at lines 127 and 173 need to be updated to align with the new type constraint

Important Files Changed

File Analysis

Filename	Score	Overview
langfuse/experiment.py	2/5	Removes None from Evaluation.value type hint and docs, but examples still use value=None, creating contradictions

Sequence Diagram

sequenceDiagram
    participant User
    participant Evaluator
    participant Evaluation
    participant LangfuseAPI
    
    User->>Evaluator: Call evaluator function
    Evaluator->>Evaluator: Process output
    alt Has expected_output
        Evaluator->>Evaluation: Create with value (int/float/str/bool)
        Note right of Evaluation: value cannot be None (after PR)
    else No expected_output
        Evaluator->>Evaluation: Previously: value=None
        Note right of Evaluation: Issue: Examples still use None<br/>but type hint now rejects it
    end
    Evaluation->>LangfuseAPI: Submit score
    LangfuseAPI-->>User: Result recorded

Loading

[greptile-apps]

Additional Comments (2)

1. langfuse/experiment.py, line 127 (link)

   logic: This example now violates the type hint on line 188, which no longer accepts None as a valid value. The example should either handle the missing expected output differently or be removed.

   Or update to return a valid value instead of None.

2. langfuse/experiment.py, line 173 (link)

   logic: This example contradicts the updated type hint (line 188) which no longer allows None values. Since evaluation values cannot be None, this example should either return a sentinel value, skip the evaluation, or handle the error differently.

   Or use a different approach that doesn't use value=None.

_{1 file reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}