From bad to good bibliometric studies

by Corrado Cuccurullo & Massimo Aria 

This post is inspired by the following paper: González-Alcaide, G. (2021). Bibliometric studies outside the information science and library science field: uncontainable or uncontrollable?, Scientometrics, 1-34.

Bibliometric studies: an uncontainable or uncontrollable growth?

Scholars show a growing and generalized interest in bibliometrics as a research methodology across all areas of knowledge, "with the number of papers published in the past 15 years multiplying 12-fold and spreading to all areas of knowledge".

Bibliometric studies are particularly strong in areas like Life Sciences & Biomedicine. However, bibliometric publications are also emerging in areas that do not have traditional links with this methodology, for example, the Arts and Humanities.

This seemingly “uncontainable” growth takes place uncontrollably. For this reason, it is necessary to "stimulate reflection".

The light and dark side of the bibliometric boom 

The growth started in the mid-2000s when bibliometrics has been increasingly integrated into researcher assessment and research activity management.

It has also become easier to access a large volume of bibliometric metadata, the field has known several methodological advances, and many different software tools for analyzing and visualizing bibliometric data were developed. The following table is provided by Muñoz et al. (2020). Their conclusion is:

For software tools, please, see also Bale et al. (2020)

Anyway, "this boom has emerged in a widely dispersed and spontaneous way, with the emergence of different, autonomous research components", characterized by no link with the theoretical foundation of the discipline:

• The absence of standards in the discipline and/or the appeal of "easy quantization" to academic opportunists push the observed growth.
• Academic opportunists "adopt a few basic methods or notions to quickly generate publications, to the detriment of the theoretical, interpretative, and critical foundation on which any bibliometric study should be based".
• These opportunistic behaviors are possible due to (i) the lawfulness of groups, organizations, or societies and (ii) contexts where there aren't external controls and the ability to identify and sanction inappropriate behaviors.
• These contexts generate a feeling of impunity because opportunists see potential benefits outweighing the negative consequences they can suffer.
• These opportunistic behaviors often translate into “bad bibliometrics”, which is characterized by "the publication of superficial studies with an unclear approach, objective, or critical interpretation of the results".

5 golden rules to enhance the quality of bibliometric papers

The improvement in the quality of research, along with the publication of studies that meaningfully contribute to advancing knowledge, are essential pillars sustaining the development and maturity of a scientific discipline. Some possible "golden rules" are the following:

1. Scholars who want to use bibliometrics as a research method must have or must strengthen their training in this discipline. Bibliometric laws, indicators, and tools must become the founding nuclei of the training of every PhD student. It is important that the conceptual and intellectual roots of bibliometrics become an interdisciplinary heritage. It is preferable that this training is provided by field experts.
2. Researchers using bibliometrics for science mapping purposes should refer to the guidelines for systematic literature reviews.
3. Inexperienced scholars are invited to promote research groups composed of researchers with bibliometric skills and thematic experts from other different fields.
4. Researchers - experts in bibliometrics - must provide guidelines and standards to capture the essential elements of any study based on bibliometric methods. Indeed, the great impact that DORA and the Leiden Manifesto (both on Research Assessment) have had should spur a similar claim in support of the quality of bibliometric research.
5. The review of bibliometric papers must be more rigorous and critical before the acceptance and publication of this type of study. Publishers and non-specialized journals that publish bibliometric studies must evaluate papers in accordance with the standards expected from any other research work: originality, innovation, advancement of knowledge, and methodological rigor. When bibliometrics is used for the aim of scientific mapping, the rules of systematic literature review must be respected.

The bibliometrix website will be a hub to promote training in bibliometrics, through intelligible teaching materials

References

• Bales, M. E., Wright, D. N., Oxley, P. R., & Wheeler, T. R. (2020). Bibliometric visualization and analysis software: State of the art, workflows, and best practices.
• Muñoz, J. A. M., Viedma, E. H., Espejo, A. L. S., & Cobo, M. J. (2020). Software tools for conducting bibliometric analysis in science: An up-to-date review. El profesional de la información, 29(1), 4.

Written by by Corrado Cuccurullo & Massimo Aria