STI 2024 is here!

Cinzia Daraio (Sapienza University of Rome), Juan Gorraiz (Dept Bibliometrics & Publication Strategies, University Vienna) and Wolfgang Glänzel (ECOOM, KU Leuven).

Recently, research evaluation using quantitative methods has received much criticism, both from a part of the scientific community and from recent initiatives at the European level calling for a rethinking of research evaluation by applying mainly peer-review. We focus on the use of bibliometric indicators in an evaluative context. We sketch a general framework of criteria that should be considered in the use of bibliometric indicators in order to ascertain whether their use, in the specific evaluation context, is appropriate for its intended purpose or not. Are bibliometric indicators always inappropriate, or should they be used with care and skill, with respect to the evaluative problem under consideration? In this paper we caution against “throwing the baby out with the bathwater” and advocate the idea that bibliometric indicators, even the number of publications and citations received, if used “appropriately” can still be extremely useful for research evaluation.

Luca D’Aniello (University of Naples Federico II), Nicolás Robinson-García (University of Granada), Massimo Aria (University of Naples Federico II) and Corrado Cuccurullo (University of Campania Luigi Vanvitelli).

The surge in scientific publications challenges the use of publication counts as a measure of scientific progress, requiring alternative metrics that emphasize the quality and novelty of scientific contributions rather than sheer quantity. This paper proposes the use of Relaxed Word Mover’s Distance (RWMD), a semantic text similarity measure, to evaluate the novelty of scientific papers. We hypothesize that RWMD can more effectively gauge the growth of scientific knowledge. To test such an assumption, we apply RWMD to evaluate seminal papers, with Hirsch’s H-Index paper as a primary case study. We compare RWMD results across three groups: 1) H-Index-related papers, 2) scientometric studies, and 3) unrelated papers, aiming to discern redundant literature and hype from genuine innovations. Findings suggest that emphasizing knowledge claims offers a deeper insight into scientific contributions, marking RWMD as a promising alternative method to traditional citation metrics, thus better tracking significant scientific breakthroughs

You can review the work by accessing the following link

Wenceslao Arroyo-Machado (School of Public Affairs, Arizona State University, Phoenix, AZ 85004, USA), Benjamín Vargas-Quesada (Unit for Computational Humanities and Social Sciences (U-CHASS), University of Granada, Granada, Spain), Teresa Muñoz-Écija (Independent researcher, Granada, Spain)
and Zaida Chinchilla-Rodríguez (Institute of Public Goods and Policies (IPP), CSIC, Madrid, Spain).

This research-in-progress paper explores scope and challenges of the ORCID’ adoption by Spanish individuals. A sample of 182.457 records has been retrieved from ORCID. The preliminary results highlight a significant adoption of ORCID in Spain, specifically in Spanish researchers, as the overlap with the OpenAlex database reveals a strong overlap. However, the completeness of ORCID metadata fields varies, indicating uneven usage. Our findings suggest that the development of targeted educational programs and the integration of interoperability systems, which enable automatic updating of metadata, are needed. This improvement would optimise the utility of ORCID in supporting transparent and accurate academic profiling, thereby enriching the global research ecosystem.

You can review the work by accessing the following link

Wenceslao Arroyo-Machado (School of Public Affairs, Arizona State University, Phoenix, AZ 85004, USA), Rodrigo Costas (Centre for Science and Technology Studies (CWTS), Leiden University, the Netherlands) and Adrián A. Díaz-Faes (INGENIO (CSIC-UPV), Universitat Politècnica de València, Camino de Vera s/n, 46022 Valencia, Spain).

This research-in-progress paper examines the alignment between public interest, as evidenced by Wikipedia page views, and the distribution of academic resources across various health conditions. Utilising data from Wikipedia, Wikidata, and OpenAlex, the study reveals both relevant geographical correlations and notable gaps in how diseases are addressed in academic research compared to their visibility in social media. Moreover, discrepancies in content quality on Wikipedia pages indicate potential biases in the global research agenda.
These findings underscore the importance of considering both social and academic metrics to address
misalignments and advocate for a more equitable distribution of research resources in the biomedical sciences.

You can review the work by accessing the following link

Elvira González-Salmón (Universidad de Granada), Zaida Chinchilla-Rodríguez (Instituto de Políticas y Bienes Públicos (IPP), Consejo Superior de Investigaciones Científicas (CSIC)), Gabriela F. Nane (Delft Institute of Applied Mathematics, Delft University of Technology) and Nicolas Robinson-Garcia (Universidad de Granada).

We retrieve data from Dimensions, the World Bank Open Data (WBOA) and the UNESCO Institute for Statistics (UIS) to construct a country level longitudinal dataset including the yearly number of researchers by gender. Our aim is to predict when each country will reach gender parity and which factors may influence the increase of the proportion of women in science. Here we present some preliminary findings using the ARIMA and Exponential Smoothing forecasting models, and a first attempt to look into influencing factors using Bayesian Networks. 

You can review the work by accessing the following link

Wenceslao Arroyo-Machado (School of Public Affairs, Arizona State University), Mattia Caldarulo (Department of Public Policy, Rochester Institute of Technology), Timothy P. Johnson (Department of Public Policy, Management, and Analytics, University of Illinois Chicago) and Eric W. Welch (School of Public Affairs, Arizona State University).

This paper addresses the challenge of accurately matching author profiles across the major bibliographic databases OpenAlex, Scopus, and Dimensions, using the United States as a case study. The study aims to quantify and evaluate the overlap of author profiles. The methodology includes an analysis of bibliographic records published between 2018 and 2022, matching author profiles through database-specific identifiers. It was found that although the overlap between Scopus and OpenAlex is lesser (66%-68%), the matches are more precise and exact. In contrast, Dimensions struggles with its disambiguation process, leading to a significant number of profiles that do not match precisely and instead aggregate into individual profiles from other databases, creating inconsistencies. Future research will delve deeper into these discrepancies and evaluate their implications on bibliometric analyses at the author level.

Aoxia Xiao (School of Information Management, Wuhan University) and Nicolas Robinson-Garcia (Unit for Computational Humanities and Social Sciences (U-CHASS), EC3 Research Group, University of Granada).

Interdisciplinary research has become increasingly prevalent in academia, yet it faces numerous challenges, including barriers related to disciplinary boundaries, academic norms, and authorship practices. This study explores authorship dynamics across diverse research topics to better understand how scholars contribute to interdisciplinary endeavors. Using data from PLOS Publishers and ScienceDirect comprising over 750,000 publications and 2 million authors, we examine patterns of authorship and contribution across different research topics. Our analysis reveals consistent usage patterns of Contributor Roles Taxonomy (CRediT) categories across various research topics, indicating a degree of uniformity in author contributions. Through Kmeans clustering, our analysis identifies four distinct author clusters: «Sergeants,» «Soldiers,» «Generals,» and «Field Commanders.» Each cluster represents unique patterns of publication output, topic involvement, and CRediT category usage. These findings offer insights into the complexities of interdisciplinary collaboration, providing valuable knowledge for improving collaboration strmatchinategies and advancing interdisciplinary research initiatives.

You can review the work by accessing the following link

Francois Van Schalkwyk (University of Stellenbosch), Elvira González-Salmón (Unit for Computational Humanities and Social Sciences (U-CHASS), EC3 Research Group, University of Granada), Márcia R. Ferreira (Complexity Science Hub) and Nicolas Robinson-Garcia (Unit for Computational Humanities and Social Sciences (U-CHASS), EC3 Research Group, University of Granada).

This paper explores the impact of international mobility on the distribution of tasks within coreperiphery scientific collaborations, particularly focusing on Africa. Utilizing a dataset with contributions statements from ScienceDirect, the study categorizes publications and author contributions from 2017-2023 involving African researchers. It finds significant disparities in task assignment, where authors from the global scientific periphery often perform menial roles. The paper highlights that mobility, especially international, potentially mitigates these inequalities by positioning periphery scholars in more substantial roles, enhancing their visibility and career progression. Initial results indicate that mobile researchers tend to secure prominent authorial positions and are more involved in conceptual and supervisory tasks. These findings suggest that increasing international mobility and integration into global networks could promote a more equitable distribution of intellectual labour and recognition in global science collaborations

You can review the work by accessing the following link

Julian D. Cortes (School of Management and Business, Universidad del Rosario | Engineering School, Universidad de Los Andes), Nicolas Robinson-Garcia (EC3 Research Group, Department of Information and Communication, University of Granada), Zaida Chinchilla-Rodríguez (Consejo Superior de Investigaciones Científicas (CSIC) – Instituto de Políticas y Bienes Públicos (IPP)) and Maria Catalina Ramirez-Cajiao (Engineering School, Universidad de Los Andes).

 The academic age (AA) measures a researcher’s experience in producing scientific knowledge. It has an impact on national and institutional policies, including career incentives and funding. However, there have been few studies conducted in global south countries. Our aim was to compare and refine different AA approaches for the Colombian scientific workforce. We used official national data on knowledge products and researchers’ assessment. We found that AAs for different academic activities did not show multi-collinearity. Additionally, comparing national rankings with international research career standards revealed significant disparities. This may lead to an underestimation of researchers’ capacity building in the global south under AA computation and interpretation standards. Our preliminary findings contribute to discussions about the complexity and necessity of defining career stages, with consideration for diversity and inclusiveness.

You can review the work by accessing the following link

Nicolas Robinson-Garcia (University of Granada), François van Schalkwyk (Centre for Research on Evaluation, Science and Technology (CREST), Stellenbosch University), Mayra M. Tirado (School of Public Affairs, Arizona State University), Victoria Pham (School of Public Affairs, Arizona State University) and Julia Melkers (Arizona State University)

This study explores the conceptualization of ‘team science’ within the field of scientometrics, identifying key attributes that define scientific teams through a systematic literature review and AI-assisted analysis. We examine definitions and operationalizations from 26 pivotal studies, synthesizing a unified definition centred on three main attributes: interdisciplinary composition, shared goals, and collaborative effort. Our findings highlight the diversity and inconsistency in current descriptions, suggesting a need for a more standardized framework. This paper contributes to the discourse on team science by proposing a refined, empirically testable definition aimed at enhancing comparative studies and improving team dynamics in scientific research.

You can review the work by accessing the following link

Julia Melkers (Arizona State University), Nicolas Robinson-Garcia (INGENIO (UPV-CSIC) Universitat Politècnica de València) and Luyu Du (Arizona State University).

Academics are no longer isolated scientists and engineers. Collaboration is the norm of science, and the co-production of academic research with non-academics is recognized as critical to producing research that can lead to broader impacts. However, the dynamics and outcomes of such collaborative efforts, particularly in terms of co-production of new knowledge, remain less understood. This study investigates the publication patterns of academic scientists collaborating with non-academic partners across various sectors. We use data from a robust national survey and related lifetime bibliometric data to quantitatively analyze these collaborative patterns. A notable aspect of our work is that we categorize collaborative ties by research and non-research, as well as by sector. The implications for the results of this study are relevant to the growing body of work on coproduction of knowledge and will contribute to the scientific and technical human capital model.

Nicolas Robinson-Garcia, Gabriela Nane, Gemma Derrick

This special session will serve as an exploratory forum to critically examine and envision the future of research evaluation in the context of sustainability, technological integration, and societal impact. Recognizing the evolving and somewhat elusive concept of sustainable scholarship, this session is designed to open up a multifaceted dialogue and brainstorm potential research direc tions that can lead to actionable changes. We will commence with two brief, thought – provoking presentations that set the stage by highlighting key challenges and opportunities within sustainable scholarship. The focus will be particularly on the integrati on of Artificial Intelligence (AI) and the dynamics of peer review systems. 

More specifically we will address the following challenges:

1. Design and Modelling of Evaluation Panels: Addressing biases related to gender, ethnicity, ageism, and other forms o f discrimination.
2. Biases in Evaluative Processes by Experts and Metrics: Critically examining the inherent biases present in the processes conducted by experts and research metrics.
3. Efficiency and Responsibility in AI Utilization: Discussing the role of AI in maximizing research evaluation efforts and the ethical considerations tied to its use.
4. Diversity, Creativity, and Academic Freedom: Encouraging environments that foster diversity and creativity, ensuring that academic freedom remains at the fo refront of scientific inquiry. 

Following the presentations, the floor will open for a discussion, inviting all participants to brainstorm and contribute ideas and perspectives on shaping a sustainable research evaluation culture. Participants will engage in a hands – on experiment involving expert judgment. They will be presented with a scenario that requires the selection of candidates for a hypothetical academic position, aiming to expose and address inherent biases and decision – making processes. This sess ion will serve not only as a discussion platform but also as an incubator for innovative ideas that can drive a more equitable and impactful research evaluation ecosystem.