Skip to content
@prometheus-eval

prometheus-eval

Codebase to inference and train foundation models specialized on evaluating other foundation models
README.md

We train language models specialized in evaluating other language models and optimize evaluation pipelines!

Repositories

Below are our key projects, with links to their repositories and related publications:

Repository Description Paper
prometheus-eval A repository for evaluating LLMs in generation tasks. Supports Prometheus 2, GPT-4, and others. Link
prometheus An Evaluator LM that is open-source, offers reproducible evaluation, and inexpensive to use. Link
prometheus-vision An Evaluator VLM that is open-source, offers reproducible evaluation, and inexpensive to use. Link

Popular repositories Loading

  1. prometheus-eval prometheus-eval Public

    Evaluate your LLM's response with Prometheus and GPT4 💯

    Python 1.1k 68

  2. prometheus prometheus Public

    [ICLR 2024 & NeurIPS 2023 WS] An Evaluator LM that is open-source, offers reproducible evaluation, and inexpensive to use. Specifically designed for fine-grained evaluation on a customized score ru…

    Python 327 19

  3. prometheus-vision prometheus-vision Public

    [ACL 2024 Findings & ICLR 2024 WS] An Evaluator VLM that is open-source, offers reproducible evaluation, and inexpensive to use. Specifically designed for fine-grained evaluation on customized scor…

    Python 86 6

  4. cmu-paper-reviewer cmu-paper-reviewer Public

    Code repository for the "CMU Paper Reviewer System", a agentic system that generates reviews for academic papers.

    Python 28 2

  5. K-BrowseComp K-BrowseComp Public

    Repository for "K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts"

    Python 16

  6. scaling-evaluation-compute scaling-evaluation-compute Public

    Repository for "Scaling Evaluation-time Compute with Reasoning Models as Process Evaluators"

    12

Repositories

Showing 10 of 10 repositories
  • national-lab-bench Public

    Official code repository for "Shoulders of Giants: Benchmarking AI agents on open scientific research problems"

    prometheus-eval/national-lab-bench's past year of commit activity
    Python 0 Apache-2.0 0 0 0 Updated Oct 6, 2026
  • cmu-paper-reviewer Public

    Code repository for the "CMU Paper Reviewer System", a agentic system that generates reviews for academic papers.

    prometheus-eval/cmu-paper-reviewer's past year of commit activity
    Python 28 Apache-2.0 2 0 0 Updated Sep 2, 2026
  • K-BrowseComp Public

    Repository for "K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts"

    prometheus-eval/K-BrowseComp's past year of commit activity
    Python 16 MIT 0 0 0 Updated Jun 2, 2026
  • prometheus-eval Public

    Evaluate your LLM's response with Prometheus and GPT4 💯

    prometheus-eval/prometheus-eval's past year of commit activity
    Python 1,119 Apache-2.0 68 12 1 Updated Apr 25, 2025
  • scaling-evaluation-compute Public

    Repository for "Scaling Evaluation-time Compute with Reasoning Models as Process Evaluators"

    prometheus-eval/scaling-evaluation-compute's past year of commit activity
    12 0 1 0 Updated Mar 25, 2025
  • prometheus-vision Public

    [ACL 2024 Findings & ICLR 2024 WS] An Evaluator VLM that is open-source, offers reproducible evaluation, and inexpensive to use. Specifically designed for fine-grained evaluation on customized score rubric, Prometheus-Vision is a good alternative for human evaluation and GPT-4V evaluation.

    prometheus-eval/prometheus-vision's past year of commit activity
    Python 86 Apache-2.0 6 3 0 Updated Sep 13, 2024
  • .github Public

    Organization README for prometheus-eval

    prometheus-eval/.github's past year of commit activity
    0 0 0 0 Updated Jun 11, 2024
  • leaderboard Public

    BiGGen-Bench Leaderboard

    prometheus-eval/leaderboard's past year of commit activity
    Python 0 0 0 0 Updated Jun 4, 2024
  • prometheus-eval.github.io Public

    Documentation and blogposts for Prometheus

    prometheus-eval/prometheus-eval.github.io's past year of commit activity
    0 0 0 0 Updated May 1, 2024
  • prometheus Public

    [ICLR 2024 & NeurIPS 2023 WS] An Evaluator LM that is open-source, offers reproducible evaluation, and inexpensive to use. Specifically designed for fine-grained evaluation on a customized score rubric, Prometheus is a good alternative for human evaluation and GPT-4 evaluation.

    prometheus-eval/prometheus's past year of commit activity
    Python 327 MIT 19 4 0 Updated Nov 11, 2023

Top languages

Loading…

Most used topics

Loading…