Everything / Evals and observability / LLM evaluation
LLM evaluation Emerging
On TutorHero's Everything Index, LLM evaluation is emerging on GitHub: 1.36 of every 10,000 new public repos over the last three months (1.3k a month), 20x vs a year ago. The 12-month forecast says rising (+113%). By the end of 2032, lines in the same state ended between 0.01x and 2.5x of where they started (median 0.54x).
Since 2019, and the range to 2032
New repos per 10,000 (3-month average, log scale)12-month forecastRange to 2032: middle half and 90% of past similar linesJev's 2032 call
The range is not a forecast of this line: it is what GitHub lines in the same state (more than doubled its share of new repos in a year) did 1 to 5 years later, 2019 to 2026, applied to today's level. Past 5 years it is stretched, not measured. Checked out of time (learned before 2022, scored after), 71% of outcomes landed inside the 90% range, so read it as wide. Open it in the Everything chart.
Week by week
New public repos with the llm-evaluation tag each week (Monday to Sunday, UTC). The outlined bars, last week included, are still filling in, as tags are often added days after a repo is created, so their height will grow a little. "Against the field" divides the change by what the typical tag did, so a wave that lifts every tag does not count. Hover for the numbers.
Built vs talked about
Each line is its own share (of all new GitHub repos, all Hacker News stories, all English Wikipedia views, all new stars on the repos we track), shown as a share of its own peak, so the timing and shape compare even though the units differ. Hover for the counts. Tap a name to hide or show a line.
What it means
- Worth learning now: LLM evaluation is emerging, 20x share of new GitHub repos vs a year ago.
- Start by reading ifixai-ai/iFixAi (+5.4k stars in 3 months) and langfuse/langfuse (+1.3k stars in 3 months).
- People to follow: Confident AI, superwesleyhys-ux, Scale.
- Jev for someone who wants to contribute or get hired: Commit this year (3.3 of 4).
- Getting crowded: new repos 20x but attention -5%, so each new project competes harder for stars.
- 4.20 new stars per new repo over 3 months, vs 8.50 for the typical trend: more supply than demand right now.
- The 12-month forecast for its share: +113% (Rising).
- Built more than talked about: Hacker News +53% vs new repos 20x in a year, less talk per builder than the typical trend, so builders are ahead of the conversation.
- Jev for a builder: Commit this year (3.0 of 4).
- History: of past GitHub lines that more than doubled its share of new repos in a year, 61% had a smaller share 3 to 5 years later and 8% at least doubled.
- Jev's 2032 call: better than most similar lines (unsure), which on the base rates means about 0.78x today's share by the end of 2032.
- Attention: 0.87% of new stars on the repos we track, -5% in a year.
- Hacker News: 12 stories a month, +53% share in a year.
- Jev for an investor: Commit this year (2.9 of 4).
- Not investment advice: this measures what people build on GitHub, not any company.
- Most-starred tools here lately: ifixai-ai/iFixAi (vouched), langfuse/langfuse (vouched).
- The field is emerging, so expect new tools and fast changes; pin versions.
- Jev for a developer picking tools: Commit this year (2.8 of 4).
Repos carrying it now
People building it
Confident AILLM evaluation and red teaming frameworks.Influence 46
superwesleyhys-uxsuperwesleyhys-uxInfluence 16
ScaleData platform for AI and AI agent environments.Influence 26
aaron-for-valueaaron-for-valueInfluence 11
White CircleRuntime AI safety and alignment infrastructureInfluence 32
ChromaSearch infrastructure for AI applications.Influence 53
chainlitBuilding conversational AI applications quickly with open-source tools.Influence 23
符荣峻白箱AGI架构、记忆基准与共同信任协议探索。Influence 35More in Evals and observability
Questions
Is LLM evaluation growing on GitHub?
Yes. Its share of all new public GitHub repos is 20x vs a year ago and +97% over the last three months (Sep 2026). Phase: Emerging.
How many new LLM evaluation repos are created each month?
About 1.3k new public repos a month carry the llm-evaluation tag (three-month average to Sep 2026), or 1.36 of every 10,000 new repos.
What will LLM evaluation look like in 2032?
Nobody knows, and we say how much. Of past GitHub lines that more than doubled its share of new repos in a year, the middle half had between 0.14x and 1.27x of their share 3 to 5 years later, and 61% ended smaller. Jev's call: Better than most similar lines.
Is LLM evaluation more talked about than built?
Built more than talked about. Over the year its share of Hacker News stories moved +53% while its share of new GitHub repos moved 20x. New repos grew faster than talk for most AI trends this year, so the label compares it with the typical trend: about 1/3.7 of the talk per builder of the median trend.
Not investment advice. Measurements of what people build and star on GitHub, a statistical forecast, base rates and a model's reading of them. Nothing here is about any company's revenue, valuation or stock.