Skip to Main Content

Graduate Research Portal: Measuring

Library-based support for University of Lethbridge graduate students

Citation Analysis, Impact Factors & Altmetrics

Citation analysis, impact factors and altmetrics are a few of the many different ways scholars are able to measure the impact, importance and viewership of their own published works and that of their colleagues. As a graduate student who may go on to teach, research and publish your findings, having an understanding of how your work will be evaluated is invaluable when it comes to obtaining and retaining a faculty or research position.

from Innovation@ETH-Bibliothek

 

Importance of Bibliometrics

Bibliometrics are a way to rate or quantify the performance and publications of individuals and academic institutions. The most common forms are citations, impact factors (IFs), and h- and c- indices, which can all be used as ways to measure the impact of a scholar's or institution's published research (Davidson et al. 1).

According to the University of Leeds, bibliometrics can enhance scholarly activities by: 

  • Demonstrating the importance and impact of your research and/or that of your research group, which can be useful for:
    • applying for tenure, promotion or grants
    • reflecting the importance of your research in your CV\
    • demonstrating the value of your research to your institution
    • demonstrating return on investment to funding bodies, industry and the tax-paying public
  • Identifying areas of research strength and weaknesses, which can be useful for:
    • informing future research priorities for an institution
  •  Identifying top performing journals in a subject area, which can be useful for:
    • deciding where to publish
    • learning more about a subject area
    • identifying emerging areas of research
  • Identifying top researchers in a subject area, which can be useful for:
    • locating potential collaborators or competitors
    • learning more about a subject area
    • informing the job-hunting process.

Sources

Davidson, P. M. (2014). Rating and ranking the role of bibliometrics and webometrics in nursing and midwifery. The Scientific World, 2014: 1-6.

University of Leeds Library. (2016). Measuring research impact: What are bibliometrics?

Citation Analysis

Citation analysis is an effective way to uncover links between works by multiple authors as well as to uncover more resources about a similar topic. From an administrative point of view, citation analysis can be used to determine the value, i.e., quantify, a scholar's paper and its impact on the field. 

Citation Analysis Sources

Web of Science Core Collection

Searches a number of different citation analysis related databases, including:

  • Science Citation Index Expanded
  • Social Sciences Citation Index
  • Arts & Humanities Citation Index
  • Conference Proceedings Citation Index - Science
  • Conference Proceedings Citation Index - Social Science & Humanities
  • Book Citation Index - Social Sciences & Humanities

Google Scholar

Clicking the "Cited by..." button under the title of a resource provides a list of all the other indexed works that have cited the selected resource

Impact Factor

The journal impact factor is a metric used to gauge the relative importance of a particular journal relative to other journals in its field. For scholars, being published in a journal with a high impact factor is a sign of prestige. It can carry considerable weight if you eventually apply for a position at a university or research institute.

A journal's impact factor is calculated by tallying, for a given year, the number of citations to its articles published over the previous two years, and dividing that figure by the total number of articles published in the journal over the previous two years. (See the Journal Citation Reports published by Thomson Reuters.)

Advantages

As one of the oldest and most used metrics for evaluating the impact of scholarly journals, the journal impact factor has a number of different advantages when it comes to evaluating the impact, importance and significance of a scholarly journal one may wish to publish their work in. These advantages include, but are not limited to, how well known the journal impact factor is, how easy it is to calculate and how it accounts for journals of different sizes and ages.

Renown

  • Has been in use since 1950
  • Well-established and well-respected measure of the importance and significance of a journal and its articles
  • Other scholars understand the prestige of being published in a journal with a high impact factor

Simplicity

  • Simple calculation (journal's average number of per-article citations to articles published in that journal over the previous two years)

Equality

  • Does not discriminate against smaller journals that publish infrequently or more mature journals
  • A journal that publishes a handful of quality articles each year can have a higher impact factor than one that publishes many lower quality articles

Main content are

Disadvantages

Despite its popularity and widespread adoption, the journal impact factor has a number of disadvantages and flaws that scholars must be aware of, such as using self-citation to artificially boost the impact factor, publishing a large number of review articles and the "eigenlob" author self-citation phenomenon. 

Self Citation

  • Some publishers may require their authors to reference articles already published in that journal, therefore bumping up its impact factor
  • Some journal publishers may enter agreements with other publishers and require their authors to cite each other’s journals, therefore raising all of their impact factors

Review Articles

  • Journals may publish a large number of review articles that quickly and efficiently inflate their impact factor, since review articles typically receive many more citations than original articles
  • As a result, scholars who choose a journal to publish in based on its impact factor may be selling themselves short, since "60% of the top 25 journals, as ranked by the ISI impact factor, are journals publishing only reviews and summaries of past research" (Falagas & Vangelis 224)

Eigenlob Author Self-Citation Phenomenon

  • In order to take advantage of how the journal impact factor is calculated, some journals may only accept authors who publish a lot of research, which includes citing their own work
  • Journals may also prefer to publish papers that have many authors, as this increases the chance of self-citation and so increases that journal’s impact factor

Sources

Nisonger, T. E. (2004). The benefits and drawbacks of impact factor for journal collection management in libraries. The Serials Librarian 47(1): 57-75.

Falagas, M. E., & Alexiou, G. V. (2008). The top-ten in journal impact factor manipulation. Archivum Immunologiae et Therapiae Experimentalis, 56(4), 223-226.

Further Reading/Viewing

The Clarivate Analytics Impact Factor

Journal Citation Reports - Journal Impact Factor

Altmetrics

Altmetrics is a new and rapidly growing alternative approach to evaluating the impact of scholarly research. According to its founder Jason Priem, altmetrics is "the study and use of scholarly impact measures based on activity in online tools and environments". Altmetrics takes advantage of the speed and popularity of social media, news media and reference management tools to provide scholars with almost-real time feedback about who and what people are saying about their research.

Examples of data sources that altmetrics include are Twitter, Facebook, Youtube and various Blogs.

Advantages

There are a number of reasons to consider using altmetrics tools as a way to evaluate and demonstrate the impact of your work.  Potential advantages include, but are not limited to, speed and relevancy.

Speed

  • Altmetrics are generated more quickly than traditional metrics such as the impact factor, which permits the impact of your publications to be assessed much sooner.
  • A study by Gunther Eysenbach uncovered a measurable relationship between the number of tweets an article receives in the first week of publication and the eventual number of citations it gets.

Relevancy

  • Altmetrics provide broader and deeper insights into the impact of scholarly articles, researchers, universities, and other such things that lie outside the scope of traditional metrics.
  • When relevance can be measured and quantified, the system functions more efficiently and provides more evidence that research is worth supporting.
  • Altmetrics can reveal how an article or piece of research affects diverse groups such as practitioners, educators and the general public.

Sources

Eysenbach, G. (2011). Can tweets predict citations? Metrics of social impact based on twitter and correlation with traditional metrics of scientific impact. Journal of Medical Internet research, 13(4), 1-20.

Galligan, F. (2013). Altmetrics: Rethinking the way we measure. Serials Review, 39(1), 56-61.

Disadvantages

Despite the potential advantages of altmetrics in terms of quickly assessing the impact and relevancy of scholarly work, they also have a number of drawbacks, such as issues of gaming, quality and comparability.

Gaming and Quality

  • There are companies that sell Facebook likes or promote articles for a price. Similarly, Twitter followers can be purchased.
  • It can be difficult to distinguish between a low quality article that receives a lot of attention and a high quality article that receives less attention. For example, an article with 50 likes from first year university students will have a greater score than an article that receives 20 likes from professors and research experts who study that particular topic.

Comparability

  • It can be difficult to use altmetrics comparatively between different disciplines. Some disciplines are more active than others online, and some may favor particular social media tools that are used less often in other areas of scholarship and creativity.
  • Fluctuations in the popularity of social media tools can reduce the reliability of altmetrics scores. MySpace and Friendster used to very active, but have now been eclipsed by tools like Twitter and Facebook, which in turn may be overtaken by other tools in the future.

Sources

Brigham, T. J. (2014). An introduction to altmetrics. Medical Reference Services Quarterly, 33(4), 438-447.

Zahedi, Z., Costas, R., & Wouters, P. (2014). How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics, 101(2), 1491-1513.

Further Reading/Viewing

Altmetrics: a manifesto

What are altmetrics