Executive Summary: Research Metrics, GScholarLens, and Strategic Implications

1. Associate Professor Jodi Schneider’s Perspective

• Critique of Metrics: Dr. Schneider argues a disconnect of counting publications (via metrics like the h-index) from the actual goals of science.
• Systemic Design: Her core argument is to design a scientific system that works for "what society wants and needs," rather than simply refining counting mechanisms.
• Inequity of Effort: According to ther, authorship order is a poor proxy for contribution. For example, a middle author might generate a critical dataset and enable the entire study, yet would receive low "credit" under weighted systems.

2. Synthesis of Opinions on "GScholarLens"

• The Tool: GScholarLens creates a "Normalized Scholar H-index" (Sh-index) by weighting citations based on author position (100% for Corresponding, 90% for First, significantly less for Middle authors).
• The Argument For: Developer Gaurav Sharma brings up the fallacy of the traditional h-index, which inflates the status of middle authors who may contribute little. The Sh-index aims to identify "project leaders."
• The Argument Against: Critics (including Schneider) view this as a technological patch to a broken system. It arbitrarily assigns value to positions that vary wildly by field, potentially penalizing interdisciplinary collaborators who contribute essential methodology rather than text.

3. The Great Debate: "Better Metrics" vs. "Better Systems"

The academic community is currently split into two camps regarding how to handle evaluation:

• Camp A: Metric Reformers (e.g., GScholarLens)

  • Goal: Create sharper, "fairer" numbers.

  • Method: Algorithms that adjust for nuance (fractional counting, positional weighting).

  • Belief: If we measure more accurately, we fix the problem.

• Camp B: System Revolutionaries (e.g., DORA, CoARA)

  • Goal: Restore qualitative judgment.

  • Method: Abandoning journal-based metrics for individuals in favor of Narrative CVs and peer review.

  • Belief: Any single number is scientifically unsound; we must evaluate the content and impact.

4. Implications for AI in Education Research

This debate is particularly relevant to our work for two reasons:

• The "AI Inflation" Risk: As Generative AI allows for the rapid production of average-quality papers, quantitative metrics (h-index, Sh-index) will likely suffer from inflation. A system that counts papers is vulnerable; a system that evaluates narrative impact is robust.
• Interdisciplinary Risks: Our field combines Computer Science (strict authorship) with Social Sciences. Tools like GScholarLens that penalize middle authors could discourage the cross-departmental collaboration essential for AI in Education.

5. Clearing the "DORA" Confusion

In the context of EU grants and Economics, an important distinction between:

• San Francisco Declaration on Research Assessment (DORA): The voluntary commitment to stop using journal-based metrics (like Impact Factors) to judge individual researchers.
• Digital Operational Resilience Act (DORA): A binding EU regulation for the financial sector regarding cybersecurity

6. Strategic Advice for Grant Applications

Although the University of Economics Varna is not currently a formal signatory of CoARA or DORA, European evaluators (Horizon Europe) increasingly expect alignment with these principles.

Recommended Actions:

• Adopt Narrative CVs: In team bios, replace lists of Impact Factors with 3–4 sentence descriptions of specific contributions (e.g., "Developed an AI framework adopted by 3 pilot schools").
• Qualitative Impact Statements: In proposals, explicitly state that success will be measured by societal uptake and open data sharing, not just publication counts.
• Alignment Statement: Include a line in management sections stating: "This project commits to the principles of CoARA by prioritizing qualitative expert assessment and diverse outputs over quantitative metrics."

Link to the article: https://www.the-scientist.com/gscholarlens-redefines-research-impact-metrics-garners-mixed-reactions-73737
For more details please contact the Science and Research Institute at UE - Varna: https://www.ue-varna.bg/bg/p/7809/za-nas/nauchnoizsledovatelski-institut

13 дек 2025