evidently-openeval-adapter: convert evaluation DataFrames/Datasets to/from EvalPort

[adhabnr-ux]

Hi Evidently team — I maintain EvalPort, an open interchange format for portable LLM evaluation test cases, graders, suites, and results (~20 framework adapters so far, including LangSmith, MLflow, Ragas, Haystack). Built and shipped a standalone adapter for Evidently: evidently-openeval-adapter.

import pandas as pd
from evidently_openeval_adapter import to_openeval, from_openeval, evaluation_result_to_openeval
from evidently import Dataset, DataDefinition
from evidently.descriptors import ExactMatch

df = pd.DataFrame({"question": [...], "expected": [...], "answer": [...]})

suite = to_openeval(df, input_columns=["question"], expected_column="expected",
                     graders=["exact_match"], descriptor_types={"exact_match": "ExactMatch"})

df2 = from_openeval(suite)   # -> DataFrame, ready for Dataset.from_pandas()
dataset = Dataset.from_pandas(df2, data_definition=DataDefinition(),
                               descriptors=[ExactMatch(columns=["expected", "answer"], alias="exact_match")])

result_set = evaluation_result_to_openeval(dataset, descriptor_columns=["exact_match"], suite_id=suite["id"])

A design note that might be useful context: since Evidently descriptors have no fixed output column name (it's whatever alias the caller chose), grader-type inference here runs off an explicit descriptor_types mapping the caller supplies rather than name-matching — and only ExactMatch gets mapped to anything other than EvalPort's generic "custom" grader type, since it's the one descriptor with no required config params to guess at (llm_judge needs a model/prompt, semantic_similarity needs a threshold, etc. — nothing this adapter has an honest value for without the real descriptor config). Every original DataFrame column also round-trips losslessly through metadata.evidently.columns, so nothing is silently dropped converting either direction.

Tested against the real evidently package and EvalPort's own JSON-Schema-backed validators (validate_suite/validate_result_set) before shipping. Not proposing any change to this repo — just flagging it here per your CONTRIBUTING guide's recommendation to open an issue and describe the contribution. Happy to answer questions about the mapping, or adjust if something about the descriptor/Dataset shapes I've inferred from the library is off.

[LakshmirajSunilSawant]

Hey, I am interested can you please assign this issue to me.

[adhabnr-ux]

Hey @LakshmirajSunilSawant — thanks for the interest, but there's nothing to assign here: this isn't an open task on evidently's side, and I don't have permissions on this repo to assign issues anyway. The adapter is already built, tested, and shipped over in EvalPort's own repo: adapters/evidently-openeval-adapter/. This issue was just a heads-up post per evidently's CONTRIBUTING guide, not a request for a contributor.

That said, if you're interested in EvalPort itself, contributions there are very welcome — adhabnr-ux/evalport#6 is the open thread for proposing a new framework adapter (pick one that isn't covered yet and comment there), and the repo's CONTRIBUTING.md walks through the exact shape (to_openeval()/from_openeval(), tests validated against the real schema validator, etc.). Happy to help if you want to pick something up there.