Metadata Matching on Crossref

Matching funders in scholarly metadata: linking names to ROR IDs

Jason Portenoy — Mon, 27 Apr 2026 00:00:00 +0000

In April 2025, we launched the metadata matching project, in order to add missing relationships to the scholarly metadata. We will do this by consolidating all existing and planned matching workflows, which enrich member-deposited metadata in Crossref. This unified service will result in a more complete research nexus. In this blog post, we share our latest milestone: developing and evaluating a strategy for matching funder metadata to Research Organization Registry (ROR) identifiers.

Key takeaways

Funder matching links funding organisation names to persistent identifiers, helping us understand how research outputs are funded and supported.
We built a new strategy to automatically match funder names in Crossref metadata to ROR identifiers.
Evaluated on a manually labeled dataset of 3,505 funder names, the strategy achieves 99% precision and 81% recall.
This is the first production deployment of Crossref’s new metadata matching framework, paving the way for future matching tasks across affiliations, references, grants, and more.

We did a brief demonstration of the funder matching process at our Community Update Call on 13th May 2026. You can watch a recording thereof on Youtube.

Introduction

In our recent blog post on metadata enrichment, we described the different ways that Crossref metadata can be enriched after its initial deposit, leading to a more complete research nexus. In this model, we can think of the metadata records served through the Crossref API as a result of several layers of enrichment applied on top of the initial deposit from a Crossref member. These layers may include member updates, community feedback, automated matching, and third-party datasets.

Metadata matching (layer 3) is when we use automated strategies to find missing relationships between entities within the scholarly record, such as relationships between research outputs, funding organisations, and grants, based on the unstructured information already present in the metadata. Our matching project aims to create a dedicated, consolidated metadata matching workflow that will eventually replace all existing production matching processes, with results made available through the REST API. We have identified the first six matching tasks that we’d like to tackle: funder name matching, bibliographic reference matching, preprint matching, affiliation matching, grant matching, and title matching.

Funder matching is a task of automatically finding an identifier of a funding organisation based on its name. Funder matching, when done well, improves the coverage and reliability of funding metadata, and the relationships between funding organisations and research outputs in particular. These relationships are critical for understanding how research is supported, tracking compliance with funder mandates, and enabling analyses of research investment.

Funder matching, as any type of matching, is not trivial because data can be noisy: the same organisation may appear under many variants, abbreviations, or translations, and some names are genuinely ambiguous. Our goal was to develop a matching strategy that results in a lot of additional identifiers while maintaining high quality of the results.

As part of this project, we will be switching the target identifier set for funder matching from the Funder Registry to the ROR registry, in line with our long-term plan to replace the Funder Registry with ROR. ROR provides an open, community-governed identifier system that is already used for affiliations and research institutions. It has become a well curated and widely-trusted catalog of organisations around the world involved in research, and it is very well suited to be the primary identifier for funders in Crossref. We are taking this opportunity to make a major move toward using ROR IDs.

This blog post describes the funder matching strategy we’ve developed and presents an evaluation of its performance, along with a new evaluation data set.

Overview of the funder matching strategy

At a high level, the funder matching strategy takes a funder name string from Crossref metadata as input and returns zero or one ROR IDs. While funder strings can occasionally map to more than one ROR ID, this strategy can only return at most one match per input string. Future versions of the strategy will allow for multiple matches.

The new matching strategy is based on the “single search” strategy previously developed at Crossref to match affiliation strings to ROR IDs, which is currently implemented in ROR’s API and which we plan to use to enrich affiliation metadata for works in Crossref. Funder matching and affiliation matching are similar tasks—they share the same target identifier set (ROR IDs), and they both use free-form text strings as their primary inputs. Most of these text strings are in English, so the strategy is optimized for English text; but the matching still works well on text in other languages, thanks in large part to ROR’s comprehensive catalog of multilingual alternate names.

However, there are also some differences in the way that these input strings tend to look across the two different tasks, so the strategy was adapted and refined specifically for funder matching. For example, affiliation strings are often much longer and contain information such as academic department and city/country in addition to the name of the institution; funder strings are usually more concise, which can often make it easy to identify an exact match in ROR, but requires more extensive exclusion criteria to prevent incorrect matches for generic names.

The flow chart diagram shows the basic steps that each funder name goes through when a match is attempted:

Flow chart diagram of the matching strategy’s steps to evaluate a funder name against potential ROR matches

After normalization, the name is compared to a list of country names and identifiers to identify if there is any country information. The name is then passed to a search engine—an indexed text-based search system such as Elasticsearch or OpenSearch—to retrieve a set of 15-20 possible candidates of ROR organisations with similar names. At this point, we use a set of filters to discard any name matches that are unlikely to be correct (i.e., they tend to produce false positives). Some examples include matches for very short names, or names that are very generic (think “Department of Education,” without any other indication of which larger entity it may be a department of).

At this point, we have a set of candidate ROR IDs, with a corresponding set of organization names that may match our funder name. We score these names by their similarity to the input name (using a fuzzy matching algorithm), then select the best candidate based on this score and a few other heuristic measures. As a final step, we ensure that, if we identified any country information in the early stages of the matching, the ROR ID that we matched is consistent—while developing the strategy, we learned that failure to do this would be a significant source of false positives.

A core principle of the matching strategy is that it is relatively conservative: at several points in the pipeline, the strategy can explicitly abstain and return no match. This prioritizes precision over recall; we consider incorrect matches to be more harmful than missing ones. Nevertheless, this strategy will be able to fill in large gaps in the funder data, and we can be confident that we will not be making widespread mistakes. To verify this, we use an evaluation dataset, which is described in the next section.

Evaluation dataset for funder matching

To evaluate the funder matching strategy, we manually labeled an evaluation dataset that maps funder name strings from Crossref metadata to zero, one, or multiple ROR IDs. The funder names were extracted from a July 2025 snapshot of Crossref works metadata, which contains 25.7 million funder entries across 12.4 million works, representing just over 3 million unique funder name strings.

The distribution of funder names is highly skewed: a small number of names appear very frequently, while most appear only a handful of times. Because correct handling of common funders has a disproportionate impact on overall metadata quality, the evaluation dataset is a weighted sample, where each name is weighted by how often it appears without an asserted funder ID.

The final evaluation dataset contains 3,505 funder names, with a total weight of just over 2.1 million funder entries. Each name was manually labeled against the ROR registry, resulting in at least one ROR match for 1,895 names. In addition, for some cases, alternate matches were recorded to support “relaxed” evaluation in ambiguous scenarios.

Evaluation methodology

Evaluation is done by running the matching strategy on all names in the dataset and comparing the results to the manual annotations. The primary metrics are precision, recall, and the F0.5 score, which combines precision and recall while weighting precision more heavily. This reflects the project’s preference to avoid incorrect metadata assertions, even at the cost of lower recall.

In addition to standard (strict) evaluation, the framework supports relaxed evaluation using alternate matches. This is meant to address cases where funder strings might be ambiguous even for a human evaluator, or a matching strategy might identify a parent organisation of a target, which is not an entirely incorrect match.

Evaluation is performed along two independent dimensions. First, results can be calculated in an unweighted mode, where each funder name is treated as equally important, or in a weighted mode, where names are weighted by how frequently they appear without an asserted identifier in Crossref metadata. Second, evaluation can be strict or relaxed, depending on whether only the primary annotated ROR ID is considered correct or whether alternate, manually annotated matches are also accepted. Together, these dimensions produce four possible evaluation modes.

Results

Under relaxed, weighted evaluation, the funder matching strategy achieves a precision of 0.99, recall of 0.81, and an F0.5 score of 0.95.

The table below compares the performance of the matching strategy across four evaluation modes. The Relaxed Weighted mode represents the headline performance (Precision: 0.9897) as it accounts for both the frequency of names in the metadata (weighting) and valid metadata ambiguity (alternates). In practical terms, the results mean that when the strategy produces a match, it is correct (or acceptably close, in cases of genuine ambiguity) roughly 99% of the time.

Evaluation Mode	Precision	Recall	F0.5 Score	False Positives	False Negatives
Unweighted	0.9365	0.6024	0.8430	81	788
Weighted	0.9776	0.7948	0.9346	81	788
Relaxed Unweighted	0.9707	0.6445	0.8815	37	675
Relaxed Weighted	0.9897	0.8094	0.9475	37	675

While precision and recall are essential for understanding matching performance, there are other important considerations that also matter in practice. This strategy also scores high marks in some of these other criteria that we’ve identified:

Openness — The strategy is open source—source code here—and built on open source methods, in accordance with our commitment to POSI.
Explainability/Flexibility — This is not a black box machine-learning model; the steps, detailed in the overview I’ve given earlier, are fairly easy to understand, update, and apply to new data.
Resources/Speed — The strategy is very quick (averaging a matter of milliseconds per match), and does not require large amounts of intense computation or data storage.

From evaluation to production

This work represents more than an isolated matching experiment: it is intended to be the first production deployment of the new metadata matching framework. Bringing funder matching into production will involve not only implementing the strategy described here, but also standing up shared infrastructure for monitoring, iteration, and reuse across future matching tasks. Applying this new matching system across all of Crossref’s current and future funder data will be our next milestone in the project. Beyond that, we will move on to grants, affiliations, references, and more. The work we’re doing now of setting up infrastructure, refining evaluation methods, and working out any kinks as they arise, will all contribute to the momentum of the project. We’re very excited about all the enrichment of the research nexus that lies ahead!

On metadata enrichment

Dominika Tkaczyk — Thu, 19 Mar 2026 00:00:00 +0000

Metadata is communication; it can tell a story about research and paint a picture for others to respond to and learn from, across the world and throughout the forthcoming generations. Metadata can feel technical with words like ‘infrastructure’ and ‘schema’, and sometimes, like tech in general, it comes with hyperbole. But metadata really is part art (storytelling and pictures) and part science (structured models and standards) with both aspects being equally important, and requiring people as well as systems. That necessary combination of human and machine involvement also makes metadata challenging.

Crossref, as the earliest adopter of DOIs specialising in scholarly research, became synonymous with DOIs in this community. However, not everyone realises that DOIs can be registered with any one of nine different agencies, which are all separate organisations with entirely separate systems that do not at present integrate or connect. And what’s more – there isn’t a central or shared “DOI schema” – each agency develops the metadata for the purposes of their organisation or community. In Crossref’s case, with our vision to create the research nexus as a complete and robust network of relationships between objects, people, and institutions of scholarship – that community encompasses the whole of the research enterprise.

The immense 180 million records of research outputs in Crossref are maintained in a system that 24,000 member organisations have already invested in. Those records benefit from rich and format-appropriate metadata schema, developed in close collaboration with the community, which makes it possible for our members to offer contextual information about each object they register. We have a long history of working with our members on recording that context, creating tools, and providing support to adopt standard metadata, enriching the context for the benefit of the scholarly community, and society at large.

Of course, those metadata records are not perfect, both in terms of quality and completeness, and the frustration around gaps in metadata is particularly strong. We are working to improve the quality and completeness of the metadata from many angles: by working with the community to understand their needs and obstacles, by identifying and analysing potential sources for additional metadata, by maintaining and adopting the existing system to changing environment, and by planning a new flexible system that will allow third-party assertions and automated enrichment workflows.

In 2020, we published a paper for the inaugural issue of Quantitative Science Studies on Crossref: The Sustainable Source of Community-Owned Scholarly Metadata and blogged an introduction to it under Crossref Metadata for Bibliometrics. One of the things our analyses in 2019 showed was that over 80% of records between 2013-2016 had been updated. Reviewing the numbers recently, we continue to see this stewardship and maintenance of metadata, amounting to almost 70% of records from the past decade being updated at least once. On the dawn of reaching 2 billion citation links, we’d like to share our experience, plans, and views on this ubiquitous activity of updating and connecting metadata – by our members and by automations built into the system by us. Altogether, these constitute the enrichment process to improve the usability of the information for the community.

Metadata available through Crossref

Crossref collects, processes, stores, and shares metadata records for a wide range of research outputs. While each record describes an individual research output, it also mentions other entities and their attributes - and, most importantly, the relationships between them. Two works identified by DOIs, for example, may be linked by a citation relationship. A person identified by an ORCID may be connected to an institution identified by a ROR ID through an affiliation relationship. A preprint and its corresponding journal article, each with its own DOI, can be linked by an “is preprint of” relationship. A research output may be associated with a grant through a “financed by” relationship. Together, these entities and relationships form the foundational building blocks of the research nexus.

As of March 14, 2026, the Crossref database contains 180,034,490 metadata records describing research outputs. You can download all the records and examine them yourself in the latest public data file. The plot below illustrates how the number of works has changed over time, showing that the rate of growth is accelerating.

The metadata records describe research outputs of various types, including:

journal articles
books and book chapters
conference proceedings
peer reviews
reports
datasets
preprints
dissertations
grants
and more

The majority of works in the Crossref database (67%) are journal articles. However, the distribution of record types has changed considerably over time. Newer types, such as components, datasets, and posted content, are growing more quickly than more traditional ways of communicating research:

Research outputs in the Crossref database are represented by rich metadata records, which may include:

basic bibliographic metadata (title, publication dates, contributors, journal title, conference name, volume and issue numbers)
authors’ affiliations and ORCID identifiers
abstracts and links to full text
funding metadata, including funders and grant details
license metadata
bibliographic reference lists
clinical trial numbers
updates such as corrections or retractions
relationships between works and other entities, such as “is translation of”, “is review of”, “is preprint of”, or “is version of”
components associated with the work, such as figures, tables, and supplemental materials

All metadata is freely available through the Crossref REST API, and additional services, such as Crossref Search, are also provided.

A natural question is: where does all this metadata come from? This is important for two main reasons. First, it helps address the question of trust, as understanding the origin of the metadata allows users to better assess its reliability. Second, it points us to the right place when investigating or addressing issues or gaps in the data.

At first glance, the answer might seem straightforward: from Crossref members. Crossref members, such as publishers, research institutions, universities, funders, museums, libraries, data and subject repositories, and conference providers, register metadata for the outputs they publish. Crossref stores this metadata and makes it available to the community.

In reality, however, the story is more complicated.

Metadata enrichment layers

The initial metadata deposit is only the beginning of what can become a long and rather fascinating journey. What users can see in our REST API is often the result of a series of updates and additions that occur over time, sometimes coming from multiple sources and happening in different ways. We can think of these ways as enrichment layers.

Each enrichment layer offers opportunities to improve the metadata while also introducing its own considerations and challenges. Rather than forming a sequence of clearly separated stages, these layers intertwine, overlap, and affect one another, collectively shaping how a research output is represented within the research nexus.

Enrichment layers are essential for completeness of the research nexus. If we relied solely on the original, one-off deposits from members, the metadata would be full of gaps, limiting the usefulness of any analysis or assessment based on it. While the scholarly metadata will never be perfectly complete, applying these enrichment layers is how we gradually and collectively build a fuller, more accurate picture of the research nexus.

One important caveat is that more metadata doesn’t magically equal better metadata. In fact, there’s often a delicate tradeoff between completeness and quality: the harder one pushes to fill every gap, the greater the chance of introducing errors. At Crossref, we believe quality comes first. We recognise that no dataset will ever be perfect, but we’re equally unwilling to apply enrichment processes without quality control. Any enrichment we introduce must meet a high bar for accuracy — no exceptions, no shortcuts.

The order of the enrichment layers discussed here loosely reflects how established they are within the scholarly ecosystem. There also might be a correlation, or at least a perceived one, between this ordering and the reliability of the underlying processes. That said, one must tread carefully when making such interpretations: perceived reliability is not the same as actual reliability.

Layer 1: Member updates

Crossref members not only deposit metadata, but also update it over time. This is an essential part of the system for several reasons. There may be errors in the originally deposited metadata that need to be corrected. Also, the initial record may contain gaps that can be filled later as more information becomes available. In addition, many changes naturally occur: landing page URLs may change, works may be archived in new locations, or identifiers for affiliated organisations may become available. Those situations also ideally result in an update.

This update process is well established. Over 24,000 Crossref members form a large global community that operates under shared membership terms. As part of these terms, members are responsible for maintaining and updating their metadata records. In this governance framework it is clearly defined who owns and stewards the metadata associated with each record, and who is responsible for the quality level and issues.

Member updates are very common. As an example, over 80% of works deposited between 2013 and 2020 were updated at least once. This demonstrates the community’s commitment to improving completeness and quality of the scholarly record. The plot below shows the percentage of works created in a given month that were updated at least once.

However, this layer also comes with challenges. It relies on members actively meeting their obligations to maintain and improve their metadata. As a result, gaps and inconsistencies can remain, and overall metadata quality is never perfect.

Our plans for the future in this area largely build on what is already happening. This includes developing and maintaining effective user interfaces for updating metadata, evolving the input metadata schema to keep pace with changes in the scholarly landscape, offering regular workshops on metadata improvements, and collaboratively establishing best practices while educating members on how to apply them.

Layer 2: Community feedback loop

Crossref metadata is widely used and examined by a large community of consumers. As a result, issues with metadata are sometimes identified by community members and reported back to us. When this happens, Crossref does not directly correct the metadata records. Instead, we contact the relevant member responsible for the record and able to deposit an update.

In this layer, the stewardship of metadata remains with the member, while responsibility for metadata quality broadens to include other actors in the community. This creates significant potential for scaling by involving a large community in identifying and reporting metadata issues.

At present, however, this process is not automated. Crossref staff effectively act as intermediaries between those reporting issues and the responsible member. As a result, the process has limited scalability. It also depends on the willingness of members to act on the reports they receive, as they are not obligated to respond to such reports.

In the future, we may explore automating portions of this workflow to handle community feedback more efficiently and lighten the load on everyone involved.

Layer 3: Metadata matching

Metadata matching is the task of finding an identifier for an item based on a structured or unstructured description of it. Matching strategies run as fully automated processes that analyse information deposited and updated by members and add identifiers, filling gaps in the metadata.

There are many instances of metadata matching problems, for example:

bibliographic reference matching: finding a DOI for a cited paper based on a bibliographic reference,
funder matching: finding the ROR ID for a funder based on its name,
affiliation matching: finding the ROR ID for an organisation based on an affiliation string,
preprint matching: finding the DOI for a preprint that precedes a given journal article,
grant matching: finding the grant DOI based on an award number and a funder name.

This layer is unique, as it focuses on a crucial type of gap in the scholarly record: the missing relationships between entities. Indeed, adding an identifier for an entity mentioned within a metadata record of a research output is typically an equivalent of asserting a relationship between that output and the matched entity. For example, bibliographic reference matching inserts citation relationships, and funder name matching - funding relationships between a research output and a funding organisation. These relationships form the foundation of the research nexus.

Currently, at Crossref, we perform two types of matching. We match bibliographic references to the DOIs of cited outputs, and funder names to Funder IDs. Both processes rely on fuzzy comparisons and other heuristic approaches to identify likely matches.

In the case of bibliographic reference matching, as it turns out, more than half of the cited DOIs (1 billion) available in the Crossref database originate from automated metadata matching:

In the case of funder name matching, the distribution is very different, but the matching strategy was still able to fill in some of the gap:

Metadata matching is a particularly valuable form of enrichment for several reasons. Matching strategies can often achieve high levels of accuracy while working in a fully automated way. This makes them highly scalable and drastically reduces the need for human oversight. Their focus on relationships also strengthens the foundations of the research nexus.

At the same time, this enrichment layer presents a number of challenges.

Its most fundamental limitation to remember is that metadata matching can only fill gaps when there is at least some useful information to work with. For example, it can identify a cited document only using structured or unstructured citation data, and the funding organisation can only be identified if some funding information is available. But if citation information, or funding information, is completely absent, as is the case for 101M (56%) records and 166M (92%) records respectively, then matching simply isn’t possible.

Matching strategies can also be complex and time-consuming to research, develop, and maintain. They require additional considerations of issues such as openness, explainability, complexity, flexibility, and cost.

Perhaps most importantly, in the case of matching, it becomes less clear who is responsible for the information introduced through the matching process. This is particularly important because matching results are never perfect, meaning there is always a risk of introducing errors. The risk is further amplified by the fact that matching strategies typically operate in a fully automated, unsupervised manner. As a result, careful evaluation of matching performance, as well as maintaining accurate provenance records, becomes increasingly important.

At Crossref, we have ambitious plans in this area. We intend to rebuild Crossref’s metadata matching workflows using modern software development and data science practices. The goal is to create a dedicated, consolidated matching service that will eventually replace all existing production matching processes, with results made available through the REST API. This project will cover six matching tasks: bibliographic reference matching, funder name matching, preprint matching, affiliation matching, grant matching, and title matching. You can learn more about metadata matching at Crossref at a dedicated project page.

Layer 4: Third-party datasets

There are many databases containing scholarly data, and one way to fill gaps in Crossref member-provided metadata is to incorporate additional metadata from those external sources.

We already have one example of this. Crossref ingests data from the Retraction Watch database to supplement information about retractions and other updates to records:

This layer has several advantages. It draws on subject-specific and metadata-specific expertise, avoids reinventing work that has already been done elsewhere, and reflects a collaborative community-driven approach to improving the scholarly record.

However, there are also important challenges to consider. Integrating external data often involves multiple data licenses or acquisition arrangements, and there may be less control over data quality compared to metadata that comes directly from members. There is also a risk that relying too heavily on external sources could shift responsibility away from the member stewards of the metadata. Finally, it can be difficult to determine which external datasets provide sufficient value and longevity to justify long-term integration.

Looking ahead, we plan to explore further opportunities to incorporate third-party datasets, carefully considering the value they bring, as well as issues of licensing, sustainability, and data quality.

Layer 5: Unstructured content scraping

A significant amount of scholarly information still exists in fully unstructured forms, such as full-text PDF documents and web pages. In principle, extracting information from these sources could help fill many gaps in existing metadata.

In a lighter-touch approach, analysing full-text documents can also help verify existing metadata elements. If such a check fails, the unverified element may be removed from the record — which, perhaps counterintuitively, can also count as enrichment, since improving accuracy is every bit as important as adding new information.

There are also important challenges to consider. Extracting metadata directly from unstructured sources could substantially shift responsibility away from the original data stewards or owners, weakening the current stewardship model. The results of automated extraction may also be inconsistent or of relatively low quality. In addition, there are potential legal and rights-related concerns, particularly when processing full-text materials. Finally, developing reliable extraction methods would require substantial research and engineering effort.

For all these reasons, the practical usefulness of this approach remains uncertain, and Crossref currently has no plans to run such processes in production. We will, however, keep a close eye on emerging extraction technologies and may consider adopting them in some form if future evaluations show clear value.

Summary

Metadata is far more than a technical afterthought of the publishing process. It is the connective tissue of the scholarly ecosystem, linking research objects, people, and institutions into a coherent, navigable network. At Crossref, this takes the form of a vast and continually evolving corpus of more than 180 million metadata records, all contributing to the emerging research nexus, being built through collective community effort to help the global research community discover, interpret, and reuse knowledge effectively.

The initial metadata record deposited by members is only the beginning. Its quality and completeness can improve over time through multiple enrichment layers: member-driven updates, community feedback, automated metadata matching, and the incorporation of third-party datasets. These processes help fill gaps and strengthen the reliability of the scholarly record, all while upholding a firm commitment to accuracy and stewardship.

Taken together, these layers reflect a long-term, collaborative effort across technology developments, community participation, and responsible automation, to ensure that scholarly metadata becomes richer, more interconnected, and more useful for everyone who relies on it.

Piecing together the Research Nexus: uncovering relationships with open funding metadata

Rocío Gaudioso Pedraza — Wed, 01 Oct 2025 00:00:00 +0000

The Crossref Grant Linking System (GLS) has been facilitating the registration, sharing and re-use of open funding metadata for six years now, and we have reached some important milestones recently! What started as an interest in identifying funders through the Open Funder Registry evolved to a more nuanced and comprehensive way to share and re-use open funding data systematically. That’s how, in collaboration with the funding community, the Crossref Grant Linking System was developed. Open funding metadata is fundamental for the transparency and integrity of the research endeavour, so we are happy to see them included in the Research Nexus.

As emphasised recently by Hans de Jonge from NWO, funding metadata’s value is in the transparency of the relationships it enables. The system is powered by the collective action of the research community– including research funders – that registers open metadata with Crossref, making these relationships possible. With close to 180,000 grant records in our corpus we wanted to know how far they reach and what story they tell.

In March 2022, we developed an approach for linking grants to research outputs and analysed how many such relationships could be established. Now we’re able to present the latest dataset that contains relationships between grants and research outputs, both those deposited by Crossref members and discovered by an automated matching strategy. It includes data deposited up to the end of July 2025.

This work is part of our ongoing Metadata Matching project.

What exactly is in this new open dataset of grant<>output relationships?

The dataset contains 250,163 total funding relationships between grants and research outputs.
We welcomed a number of funders, such as the Dutch Research Council and Fonds de Recherche du Quebec, which together registered almost 27,000 grants in the past year.
It’s clear that the more grant metadata is registered the more funding relationships we can uncover.
The percentage of relationships that are registered explicitly by Crossref members providing grants IDs in funding information has grown from less than 0.1% in 2023 to 1% (modest numbers but amazing growth!).

The methodology

We created a dataset of relationships between grants and research outputs by analysing their metadata in several ways. A relationship is included in the dataset if at least one of the following conditions is met:

A relationship was explicitly deposited by a Crossref member through a finances or isFinancedBy relationship: 488 (0.2%) relationships
The research output contains the grant DOI within the award number in the funding metadata: 2,003 (0.8%) relationships
The award numbers in the grant and the research output are similar, and the associated funding organisations are either the same, or one is the sub-organisation of the other: 247,672 (99%) relationships

The dataset includes data deposited until the end of July 2025 and contains 250,163 total relationships.

The code used to generate the dataset is available in our GitLab repository.

The results

As you can see in the graph below, the number of relationships grant-research output continues to grow as the number of grants records Crossref members register with us increases.

Figure 1: Cumulative totals of grants, linked grants, research outputs, and grant–research output relationships from 2019 to 2025. Stepwise increases correspond to the addition of major funder datasets, including Wellcome (2020), OSTI (2021), JST (2022), the European Union (2022), the Austrian Science Fund (2023), and the Fonds de recherche du Québec (2025).

Looking at the numbers broken down by grant registrants we can see that the more grants registered the more relationships can be uncovered. The table below shows funders who have at least 1,000 total grants registered and for whom at least 10% of their registered grants are linked to research outputs, showing the number of relationships, grants, linked grants and linked research outputs (sorted by the percentage of linked grants), and compared with the data from the 2023 analysis (where available) to see how the uptake of open funding metadata is evolving.

Funder	Relationships		Linked research outputs		Grants		Number of linked grants		Percentage of linked grants
Funder	2023	2025	2023	2025	2023	2025	2023	2025	2023	2025
European Union	86,979	128,572	78,576	114,491	39,703	53,473	14,860	21,402	37.4%	40%
Japan Science and Technology Agency	19,549	30,728	16,265	25,003	9,923	11,866	2,609	3,900	26.3%	32.9%
Wellcome	34,254	45,596	25,720	33,783	17,547	19,929	5,238	6,206	29.9%	31.1%
American Cancer Society	50	604	49	586	380	1,162	34	277	8.9%	23.8%
American Heart Association (AHA)	40	1,040	38	935	598	2,764	30	621	5%	22.5%
Fundacao para a Ciencia e a Tecnologia	0	27,915	0	15,681	5	17,422	0	3,793	–	21.8%
Austrian Science Fund (FWF)	–	10,387	–	7,459	–	19,576	–	2,712	–	13.9%

Table 1: Comparison between data from 2023-07-31 and 2025-07-31 of a number of Crossref members registering grants. It shows the number of relationships, grants, linked grants and linked research outputs, sorted by the percentage of linked grants.

We encourage funders to join as members once they have determined the means of effective implementation of the GLS within their processes. By further analysing metadata of matched outputs, funders have the opportunity to monitor compliance with their policies and learn more about the impact of their programs.

Following through funders’ Open Science commitments

The relationships showcased above and in the recent analysis are powered by open funding metadata. Open funding metadata plays a central role in building a transparent, accountable and high integrity research environment by making visible the connections between the funding, grantees, research outputs, and their impact. Funders’ openness mandates and Open Science commitments emphasize the importance of traceability in the research process, so ensuring that the support given-whether financial or otherwise-can be systematically recorded and shared is instrumental. Openness is also part of the strategic plans of institutions such as the International Science Council, who has explicitly called for greater transparency in funding as a way to strengthen trust in science and counter misinformation. At the same time, initiatives such as the Barcelona Declaration on Open Research Information underscores the benefits of open, reusable funding metadata for monitoring, evaluation and assessment of research and researchers.

Crossref’s Grant Linking System offers funders’ a way to demonstrate a commitment to openness, modeling the standards they expect of the research community they support, while creating a more robust, trustworthy and collaborative research ecosystem.

Economy of scale: unlocking relationships with Crossref

Crossref houses millions of records, from the ubiquitous research articles and preprints, to books, peer review records, technical reports, datasets – you name it. Our members not only register, but also regularly update their metadata as new or corrected information becomes available. Our matching workflows allow us to make visible the hidden relationships and complete and improve the metadata records by adding new and reciprocal assertions.

This analysis shows the unique value of registering funding metadata with Crossref and adding an essential piece to the Research Nexus puzzle. The relationship metadata allows the funding that underpins the research process to be connected, and contextualise scattered data points, acting as an anchor that links publications, people, and other research outputs. This is made possible by the impressive number of records continuously being registered by more than 23,000 member organisations, and by the increasing availability of funding information in the system with more research funders joining in and registering their grant metadata with us.

Next steps

As we welcome more and more funders to the GLS, we, collectively, continue to complete the Research Nexus, record by record, field by field. The more awards we have in our corpus the more relationships we’ll uncover, so we’ll keep making these analyses periodically to make sure we don’t miss them.

But it is not all on us. We are working towards a vision where Crossref Grant IDs are business as usual – where funders register their awards, grantees are aware of them and share them with publishers, and those publishers share them back with us when registering their content – closing the loop organically. We continue working on making this easier. In the upcoming works schema update a specific Crossref Grant ID field will be added in the funding information, alongside Award ID (for an internal identifier).

Crucially, as the momentum of adoption among funders increases, and thousands of Crossref Grant IDs are available in the system, we are working with all members to raise their attention to the importance and desirability of funding metadata, so inclusion of that information in metadata of all works increases and consequently, the percentage of relationships asserted by Crossref members can grow.

This matching analysis is just one example of what we do to enrich metadata to highlight relationships among works, individuals, institutions, and actions. Earlier this year, we launched the Metadata Matching project, which is a major effort to rebuild our matching workflows using modern software development and data science practices. As part of the project, we plan to expose additional matched relationships between grants and research outputs in our REST API, alongside those deposited by our members. We’ll keep you updated as we go along!

Read more about metadata matching in the blog series:

Metadata matching: beyond correctness

Dominika Tkaczyk — Wed, 08 Jan 2025 00:00:00 +0000

In our previous entry, we explained that thorough evaluation is key to understanding a matching strategy’s performance. While evaluation is what allows us to assess the correctness of matching, choosing the best matching strategy is, unfortunately, not as simple as selecting the one that yields the best matches. Instead, these decisions usually depend on weighing multiple factors based on your particular circumstances. This is true not only for metadata matching, but for many technical choices that require navigating trade-offs. In this blog post, the last one in the metadata matching series, we outline a subjective set of criteria we would recommend you consider when making decisions about matching.

Openness

Matching tools come in many different shapes and sizes: web applications, APIs, command-line tools, sometimes even enchanted crystal balls showing matched identifiers emerging from a mysterious mist! No matter what form they take, an important consideration is whether the source code and all the related resources for the matching are openly available.

Matching strategies that are either closed-source, or rely on closed-source services for their matching logic, make it difficult to fully understand and explain matching processes. This lack of transparency also makes it impossible to adjust or improve the matching logic, since we cannot understand or improve code we cannot see.

Users are similarly impeded from identifying flaws or suggesting improvements to processes they are unable to examine. By blocking this community participation, we also lose the proven cycle of real-world testing, refinement, and validation that has strengthened myriad of open source projects. The cumulative impact of both minor and major community-driven refinements over time is incredibly valuable and should not be underestimated.

Using open source matching will also help build trust in the matching workflows and results. This is one reason why open source is one of the tenets of the Principles of Open Scholarly Infrastructure, adopted by Crossref, DataCite, ROR, and other organisations who build and maintain open scholarly infrastructure.

When evaluating matching strategies, we strongly recommend prioritizing those that are fully open source. This not only ensures their transparency and trustworthiness, but also allows for the kind of continuous improvement that results from this visibility and community engagement.

Explainability

In terms of our ability to understand and improve a matching strategy, using an open source model is only the first step. What typically matters most in the context of building and maintaining matching services is that we are able to understand their underlying code and have a clear model of how matches are derived from their corresponding inputs. Even if the matching code itself and all of the resources used in the matching are open, if they are poorly documented, lack reproducibility or tests, or are otherwise opaque, there is no guarantee that it will be possible to understand or improve the strategy. Striving for a high level of interpretability in our matching plays a determinative role in how well we can understand and modify our strategies in the future.

Being able to explain the behaviour of the matching will also help you to respond to and incorporate user feedback. When users encounter errors, you will be able to do things like advise them on how to modify or clean their inputs so that the results are better. Conversely, examining the behaviour of the strategy relative to user inputs and feedback can provide you with ideas for improving the matching.

Typically, heuristic-based strategies, such as those that use forms of search or string similarity measures, like edit distance, are easier to explain than, say, machine learning models. If a strategy uses machine learning, at least some internal decisions might be made by passing data through a complex network of algebraic equations. Those can be mysterious, non-deterministic, and are famous for being hard to interpret. This doesn’t mean they should be avoided entirely - we have built and use many machine-learning based tools ourselves! Instead, it is a good idea to weigh how their inherent lack of explainability could affect your ability to continue work on the strategy and respond to user needs, relative to all the available options.

Complexity

Complexity is another aspect that can greatly affect how easy it is to maintain the strategy. Complexity is related to how many different components the strategy has and how difficult they are to use and maintain. When a strategy has multiple interconnected parts, each component becomes a potential failure point that requires discrete assessment and maintenance.

Consider, for example, two different approaches to a matching strategy: one that uses a single machine learning model versus another that uses an ensemble of models. A single model requires maintaining one set of training data, a single training pipeline, and one deployment process. If the model’s performance unexpectedly deteriorates, whether because of an issue with the training data, a configuration error, or the need for additional input sanitization, the source of the problem is easier to isolate and fix.

The ensemble, by contrast, combines multiple, specialized models, each requiring its own training data, tests, updates, and deployments. If one model in the ensemble is found to reduce the performance of the strategy, the interdependence between models can cause this degradation to cascade through the entire system and undermine its overall reliability. Correcting for these errors becomes more challenging. If fixing one model’s performance requires retraining or adjusting its outputs, this could require recalibrating the entire ensemble to maintain the balance between models, identify regressions, and prevent new errors from emerging.

In general, preferring simpler strategies not only reduces operational overhead, but also makes it easier to diagnose issues, test changes, and iterate on user feedback. When problems arise, having fewer moving parts means less places to look for the root cause and fewer components that could be affected by any fixes.

Flexibility

The metadata to which we match grows and changes over time. New records are created, existing ones are updated, with schemas changing and evolving alongside. The resources that underlie our matching are also not static. The libraries we depend on may deprecate features between versions or the taxonomies we used to categorize results might undergo significant revisions. We thus rarely have the luxury of deploying a matching strategy once and using it forever without any changes. A good strategy has to be flexible enough to adapt to such changes, with this adaptation also being both technically feasible and practical to implement.

Much of this flexibility is also determined by a matching strategy’s ability to incorporate new data. Strategies that use continuously updated databases or indices can immediately match against new metadata as it appears in the system. By contrast, some machine learning-based approaches require training on target matches and can thus be limited in flexibility and face more constraints. While some models can be incrementally updated to recognize new matches, others require retraining from scratch to incorporate these changes - a process that can be both time-consuming and resource-intensive.

Paying close attention to a strategy’s flexibility and favoring this aspect, when possible, can significantly impact its long-term viability. When comparing different matching strategies, flexibility should thus be a primary concern in your decision-making process.

Resources

Matching strategies can vary significantly in their resource requirements, including things like CPU and GPU utilization, memory consumption, storage capacity, and network bandwidth. These requirements are directly related to infrastructure costs and energy consumption, so when evaluating a matching strategy, it is necessary to assess its resource demands across all phases of the matching lifecycle. This includes things like initial model training, re-training, index construction, updates and management for all aspects of the strategy, as well as the real-world processing of matching requests. It is a good idea to measure and monitor resource usage carefully in considering which strategies to use, as the best performing strategy may also be too resource intensive to run as a service or might grow to this state over time with additional utilization.

Speed

Matching strategies can operate at a wide range of speeds, from milliseconds to minutes per match. Since the overall response time of a strategy can affect both system scalability and user experience, we should always assess the strategy’s performance for different usage scenarios and scales of data. While some strategies might perform adequately with small datasets, they can also exhibit exponential slowdowns as data volume and complexity increases or as concurrent requests grow in number. We should therefore consider carefully how requirements for matching speed might evolve with increased usage, data complexity, and total anticipated growth. The fastest matching strategy might not always be the best choice if it comes at the cost of reduced accuracy or requires large amounts of resources, but unacceptable latency can make an otherwise excellent strategy unusable in practice for many use cases.

Putting it all together

The typical life cycle of developing a metadata matching strategy is as follows:

Scoping: we define the matching task, along with its inputs and outputs.
Research: we research what existing strategies are available for our task and/or we develop our own.
Evaluation: we evaluate all available strategies, internally or externally-developed, exploring all of the aspects described above.
Decision: we choose which strategy (if any) we want to use in our production system.
Production setup: we prepare the production models, indexes, and other resources needed for the matching.
Maintenance: we monitor and adapt the strategy relative to changing data, user feedback, and new resource requirements.

In practice, these phases do not happen all at once, nor in this strict order. Often we need to proceed through multiple iterations of them to arrive at the best strategy. For example, if initial evaluation of a strategy yields poor results, we might return to the research phase to investigate other strategies or refine our understanding of the task. Often, during the maintenance phase, we receive feedback from users that indicates potential areas of improvement and then pursue them with a new round of research and evaluation.

As we cycle through these phases, ideally all the aspects described in this entry, along with the results of the evaluation, would be taken into account. Of course, this means that these decisions have to be based on multiple criteria and by making trade-offs between their performance and all other considerations. In making these complex and difficult choices, it is useful to consider two primary questions:

Are any of the considered matching strategies good enough for our use case?
Out of all the considered strategies that are sufficient for our use case, which would be the best?

The first question requires us to create clear and quantifiable criteria that allow for eliminating some of the potential strategies. As we have indicated, these could include things like the strategy being open source, minimum performance baselines using measures like precision or recall, and operational thresholds, like the strategy being able to return results quickly, relative to user expectations or the volume of data to be processed. It should be fairly easy to test these requirements and eliminate any strategies that fall short of them. If the strategies are difficult to assess, that is likely a mark against them.

If no strategies meet these criteria, we have two options: either to abandon matching entirely or to reassess and relax our criteria to align with the available options. While the former is always an option, adopting a more pragmatic lens, framing in terms of potential value (or harm) to the users, might be beneficial. Sometimes we approach matching tasks with too high expectations and a dose of realism helps us to re-center our perspectives. After more consideration, you might decide that your criteria were too stringent or realize that you need to better define and decompose the tasks to fit the available options.

When multiple strategies appear viable, the selection process becomes more nuanced. When evaluating strategies across these various dimensions, we should try to avoid placing undue weight on minor performance differences. Evaluation metrics are useful estimates of performance, but do not always translate to real-world applications and changing data. In cases where a more complex strategy offers only marginal improvements over a simpler alternative, the maintenance and operational benefits of the simpler solution often outweigh small performance gains.

This concludes our series on metadata matching, where we described the conceptual, product, and technical aspects of matching and its applications. We hope this overview was instructive and helps you to make better decisions about the use of matching in your own tools and services!

How good is your matching?

Dominika Tkaczyk — Wed, 06 Nov 2024 00:00:00 +0000

In our previous blog post in this series, we explained why no metadata matching strategy can return perfect results. Thankfully, however, this does not mean that it’s impossible to know anything about the quality of matching. Indeed, we can (and should!) measure how close (or far) we are from achieving perfection with our matching. Read on to learn how this can be done!

How about we start with a quiz? Imagine a database of scholarly metadata that needs to be enriched with identifiers, such as ORCIDs or ROR IDs. Hopefully, by this point in our series this is recognizable as a classic matching problem. In searching for a solution, you identify an externally-developed matching tool that makes one of the below claims. Which of the following would demonstrate satisfactory performance?

It is a cutting-edge, state-of-the-art, intelligent-as-they-come, bullet-proof technology! All the big players are using it. You won’t find anything better!
The tool was tested on the metadata of 10 articles we authored, and many identifiers were matched.
The quality of our matching is 98%.

Okay, okay, trick question. The correct answer here is to opt for secret answer #4: “I wouldn’t be satisfied by any of these claims!” Let’s dig in a bit more to why this is the correct response.

The importance of the evaluation

Before we decide to integrate a matching strategy, it is important to understand as much as possible about how it will perform. Whether it is used in a semi or fully automated fashion, metadata matching will result in the creation of new relationships between things like works, authors, funding sources, and institutions. Those relationships will then, in turn, be used by the consumers of this metadata to guide their understanding and perhaps even to make important decisions about those same entities. As organisations providing scholarly infrastructure, we must therefore take it as our paramount responsibility to understand any caveats or shortcomings of the scholarly metadata we make available, including that resulting from matching.

Proper evaluation is what allows us to do this, as it is impossible to know how well a given matching strategy will perform in its absence. This is true no matter how simple or complex a matching strategy may seem. Complex methods can be tailored to data with specific characteristics and might fail when faced with something different from this. Simple methods might be only appropriate for clean metadata or a narrow set of use cases.

Beyond complexity, matching strategies themselves vary widely in character, inheriting biases from their design, training data, or how a problem has been formulated. Some prioritise avoiding false negatives, while others focus on minimising false positives. Even a generally high-performing strategy might not be perfectly aligned with your specific needs or data. In some cases, the task also itself might be too challenging, or the available metadata too noisy, for any matching strategy to perform adequately.

Evaluation is, again, how we understand these nuances and make informed decisions about whether to implement matching or avoid it altogether. By now, it should also be clear that the notion “we don’t need to evaluate” is far from ideal! Given its importance, let’s explore how evaluation is actually done.

Evaluation process

In general, a proper evaluation procedure should follow the following steps:

Preparation of an evaluation dataset containing many examples of matching inputs and the corresponding expected outputs.
Applying the strategy to all inputs from the dataset and recording the responses.
Comparing the expected outputs with the outputs from the strategy.
Converting the results of the above comparison into evaluation metrics.

From this accounting, we can see that there are two primary components for the evaluation process: an evaluation dataset and metrics.

Evaluation dataset

It’s useful to conceive an evaluation dataset as the specification for an ideal matching strategy, describing what would be returned from our forever-elusive perfect matching. When creating such a dataset, what this means in practice is that it should contain a number of real-world, example inputs, along with the corresponding ideal or expected outputs, and that all data should be in the same format as the strategy is expected to process. The outputs should themselves also confirm the strategy’s overall requirements, for example, by being consistent with its cardinality, meaning whether zero, one, or multiple matches should be returned and under what circumstances. In terms of size, it’s generally useful to calculate the ideal number of evaluation examples using a sample size calculator or using standardised measures, but as a quick rule of thumb: less than 100 examples is probably insufficient, more than 1,000 or 2,000 is generally acceptable.

It is also important that the evaluation dataset be representative of the data to be matched in order to ensure reliable results. Using unrepresentative data, even if convenient, can lead to biassed or misleading evaluations. For example, if matching affiliations from various journals, building an evaluation dataset solely from one journal that already assigns ROR IDs to authors’ affiliations might be tempting. The data, having been already annotated, allow us to avoid the tedious work of labelling, and we might even know that it is produced by a high-quality source. This is still, unfortunately, a flawed approach. In practice, such datasets are unlikely to represent the entire range of affiliations to be matched, potentially leading to a significant discrepancy between the evaluated quality and the actual performance of the matching strategy, when applied to the full dataset. To assess a matching strategy’s effectiveness, we have to resist shortcuts and instead do our best to create truly representative evaluation datasets to be confident that we’ve accurately measured their performance.

Evaluation metrics

Evaluation metrics are what allow us to summarise the results of the evaluation into a single number. Metrics give us a quick way to get an estimation of how close the strategy was to achieving perfect results. They are also useful if we want to compare different strategies with each other or decide whether the strategy is sufficient for our use case, removing the need to compare countless evaluation examples from different strategies against one another.

The simplest metric is accuracy, which can be calculated as the fraction of the dataset examples that were matched correctly. While a commonsense benchmark, accuracy can be misleading, and we generally do not recommend using it. To understand why, let’s consider the following small dataset and the responses from two strategies:

Input	Expected output	Strategy 1	Strategy 2
string 1	ID 1	ID 1	ID 1
string 2	ID 2	ID 3	Empty output
string 3	Empty output	Empty output	Empty output

Both strategies achieved the same accuracy, 0.67, making one mistake each on the second affiliation string. However, a closer examination reveals that these error types are distinct. The first strategy matched to an incorrect identifier, while the second refused to return any value illustrating the limitation of accuracy as a measure: it generally fails to capture important nuances in strategy behaviour. In our example, the first strategy appears more permissive, returning matches even in unclear circumstances, while the second is more conservative, withholding them when uncertain. Although using such a small dataset would preclude drawing any definitive conclusions, it highlights how relying on accuracy alone can obscure differences in performance.

For evaluating matching strategies, we instead recommend using two metrics: precision and recall. To recap from our previous blog post:

Precision is calculated as the number of correctly matched relationships resulting from a strategy, divided by the total number of matched relationships. It can also be interpreted as the probability that a match is correct. Low precision indicates a high rate of false positives, which are incorrect relationships created by the strategy.
Recall is calculated as the number of correctly matched relationships resulting from a strategy, divided by the number of true (expected) relationships. It can also be interpreted as the probability that a true (correct) relationship will be created by the strategy. Low recall means a high rate of false negatives, which are relationships that should have been created by the strategy but were not made.

Applying these measures to our prior example, the strategies achieved the following results:

Strategy 1: accuracy 0.67, precision 0.5, recall 0.5
Strategy 2: accuracy 0.67, precision 1.0, recall 0.5

As we can see, while both strategies have the same accuracy, using precision and recall better describes the difference between the two sets of results. Strategy 1’s lower precision indicates it made false positive matches, while Strategy 2’s perfect precision shows that it made none. The identical recall scores show both identified half of the possible matches.

Of course, results calculated using such a small dataset are not very meaningful. If we obtained these scores from a large, representative evaluation dataset, it would indicate to us that Strategy 1 risks introducing many incorrect relationships, while Strategy 2 would be unlikely to do so. In both cases, we would still expect approximately half of the possible relationships to be missing from the strategies’ outputs.

Which one is more important to prioritise, precision or recall? It depends on the use case. As a general rule, if you want to use the strategy in a fully automated way, without any form of manual review or correction of the results, we recommend paying more attention to precision. Privileging precision will allow you to better control the number of incorrect relationships added to your data. If you want to use the strategy in a semi-automated fashion, where there is a manual examination of and a chance to correct the results, pay more attention to recall. Doing so will guarantee that enough options are presented during the manual review stage and fewer relationships will be missed as a result.

To get a more balanced estimation of performance, we can also consider both precision and recall at the same time using a measure called F-score. F-score combines precision and recall into a single number, with variable weight given to either aspect. There are three commonly used types, each calculated as the weighted harmonic mean of precision and recall:

F0.5: Precision is weighted more heavily. It can be understood as a score that is 50% more sensitive to precision than recall. A high F0.5 score indicates a measure of performance that minimises false positives.
F1: Equal weight is given to both precision and recall. It can be interpreted as the most balanced score in this set. High F1 indicates good overall performance, with both false positives and false negatives being minimised equally.
F2: Recall is weighted more heavily. It can be understood as a score that is 50% more sensitive to recall than precision. A high F2 score indicates a measure of performance where false negatives are minimised.

Each of these variants allows for fine-tuning the evaluation metric to align with your expectations for a specific matching task. Choose whichever reflects the relative importance of precision versus recall for your use case.

To summarise, to avoid falling prey to misleading sales pitches or silly quizzes, it is important to have a good understanding of the performance of any strategies you are building or integrating. With thorough evaluation, including a representative dataset and carefully considered metrics, we can estimate the quality of matching and, by extension, its resulting relationships.

Now that we’ve covered how to evaluate effectively, we can move on to some other aspects of metadata matching. Our next blog post will take a final, more holistic view of matching, exploring some complementary considerations to all of the preceding. Stay tuned for more!

The myth of perfect metadata matching

Dominika Tkaczyk — Wed, 28 Aug 2024 00:00:00 +0000

In our previous instalments of the blog series about matching (see part 1 and part 2), we explained what metadata matching is, why it is important and described its basic terminology. In this entry, we will discuss a few common beliefs about metadata matching that are often encountered when interacting with users, developers, integrators, and other stakeholders. Spoiler alert: we are calling them myths because these beliefs are not true! Read on to learn why.

If you have stuck with us this far in our series, hopefully, you are at least a bit excited about the possibility of creating new relationships between the works, authors, institutions, preprints, datasets, and myriad other objects in our existing scholarly metadata. Who would not want all of these to be better connected?

We have to pause for a moment and be honest with you: metadata matching is a complex problem, and doing it correctly requires significant effort. What is worse, even if we do everything right, our matching won’t be perfect. This may be counterintuitive. Perhaps you’ve heard that matching is not a hard problem, or have encountered people surprised that a matching strategy returned a wrong or incomplete answer. Sometimes, it is obvious to a person from looking at some specific example that a match should (or should not) have been made, so they naturally assume that a change to account for this has to be simple.

Misconceptions like these can be problematic. They create confusion around matching, drive users’ expectations to unreasonable levels, and make people drastically underestimate the effort needed to build and integrate matching strategies. So let’s dive right in and debunk a few common myths about metadata matching.

Myth #1: A metadata matching strategy should be 100% correct

Anyone who has built or supported a matching strategy has likely encountered the following belief: it is possible to develop a perfect strategy, meaning one that always returns the correct results, no matter the inputs. The unfortunate truth is that while one’s aim should always be to design matching strategies that return correct results, once we move beyond the simplest class of problems or artificially clean data, no strategy can achieve this outcome. In thinking through why this is the case, some inherent constraints become obvious:

The inputs to matching are often strings in human-readable formats, which can vary wildly in their structure, order and completeness. Since they’re intended to be parsed by people, instead of machines, they’re inherently lossy and frequently unstructured, anticipating that a person can infer from the source context what is being referenced. Matching strategies, although built to make sense of unstructured data, unfortunately, don’t have the luxury of this flexibility. A strategy has to account for translating a messy, partial, or inconsistent input into a correct and structured match.

Consider, for example, the following inputs to an affiliation matching strategy:

“Department of Radiology, St. Mary’s Hospital, London W2 1NY, UK”
“Saint Mary’s Hospital, Manchester University NHS Foundation Trust”
“St. Mary’s Medical Center, San Francisco, CA”
“St Mary’s Hosp., Dublin”
“St Mary’s Hospital Imperial College Healthcare NHS Trust”
“聖マリア病院”

In order to correctly identify the organisations mentioned here, the matching strategy must be able to distinguish between different ways of representing the same institution, disambiguate multiple institutions that have similar names, and handle variant forms for the parts of each name (Saint/St./St), identify the same name in different languages (“聖マリア病院” is Japanese for “St. Mary’s Hospital”), and make assumptions about partial or ambiguous locations translating to more precise references. While a person reviewing each of these strings might be able to accomplish these tasks, even here there are some challenges. Does “St Mary’s Hosp., Dublin” refer to the hospital in Ireland or a separate hospital in one of the many cities that share this name? Should we presume that because “聖マリア病院” is in Japanese, this refers to a hospital in Japan? Would someone, by default, be aware that St. Mary’s Hospital in London is part of the Imperial College Healthcare NHS Trust, such that inputs one and five refer to the same organisation?

An additional challenge lies in the quality of the data, which in the context of matching, encompasses both the input and the dataset being matched against. In real world circumstances, no dataset is fully accurate, complete, or current and certainly not all three. As a result, there will always be functionally random differences between inputs to the strategy and the entities to be matched. A theoretically perfect matching strategy would thus need to distinguish between inconsequential discrepancies resulting from gaps, errors, and variable forms of reference and actual, meaningful differences indicating an incorrect match. As one might imagine, this would require near total knowledge of the meaning and context for all inputs and outputs, a nigh-on impossible task for any person or system!

As a consequence, no metadata matching strategy will ever be perfect. It is unreasonable for us to expect them to be. This does not mean, of course, that all strategies are equally flawed or destined to forever return middling results. Some are better than others and we can improve them over time. Which brings us to the next myth:

Myth #2: It is always a good idea to adapt the matching strategy to a specific input

Matching strategies are not static. They can - and should - be improved. There is, however, a deceptive trap that one can fall into when attempting to improve a matching strategy. Whenever we encounter an incorrect or missing result for a specific input, we treat this problem like a software bug and try to adapt the strategy to work better for it, without considering all other cases.

The more complicated reality is that the quality of matching results is controlled through a complex set of trade-offs between precision and recall that determine the kind and number of relationships created between items:

Precision is calculated as the number of correctly matched relationships resulting from a strategy, divided by the total number of matched relationships. It can also be interpreted as the probability that a match is correct. Low precision indicates a high rate of false positives, which are incorrect relationships created by the strategy.
Recall is calculated as the number of correctly matched relationships resulting from a strategy, divided by the number of true (expected) relationships. It can also be interpreted as the probability that a true (correct) relationship will be created by the strategy. Low recall means a high rate of false negatives, which are relationships that should have been created by the strategy but were not made.

The diagram depicts false negatives and false positives. The ideal outcome would be that the ellipses are identical, matched relationships are exactly the same as true relationships, and there are no false negatives or false positives. In practice, we try to make the intersection as big as possible.

The tradeoff between precision and recall roughly means that modifying the strategy to improve recall will decrease precision, and vice versa.

Imagine, for example, we received a report about a relationship that was missed by matching because of a partial, noisy, or ambiguous input. We might be tempted to resolve this issue by relaxing our matching criteria. Unfortunately, this will have a cost of a higher overall rate of false positive matches.

Conversely, if we encounter a case where the matching has returned an incorrect match, we might attempt to make the matching strategy stricter to avoid this result. We should remember, however, that this may have the consequence of causing the strategy to skip many perfectly valid matches.

The tradeoff between precision and recall. (a) A strict strategy prioritises precision over recall resulting in more false negatives. (b) A relaxed strategy prioritises recall over precision resulting in more false positives.

Striking this balance becomes even more difficult when attempting to address multiple issues at once, or considering constraints like the time and resources consumed by each aspect of the strategy. Each choice can compound the individual effects in unanticipated and expensive ways. The aim of matching ultimately then can’t be to achieve perfect results for every single case. Fixing one particular situation might not be desirable, as it can result in breaking multiple other cases. Instead, we have to find a locally optimal balance that optimises the strategy’s utility, relative to these inherent limitations. This means accepting some level of imperfection as not just inevitable, but necessary for implementing a workable strategy. When you consider all this, you might conclude that…

Myth #3: We shouldn’t do large-scale, unsupervised matching

Imperfect matching strategies, when applied automatically to real-world large datasets, might:

Fail to discover some relationships (false negatives), an outcome that may not be terribly problematic. In the worst case scenario, we have wasted a great deal of effort developing matching strategies that do not improve our metadata.
Create incorrect relationships between items (false positives), what seems like a potentially larger problem, where we have added incorrect relationships to the metadata.

Many have the instinct to avoid false positives at any cost, even if this means missing many additional correct relationships at the same time. They might come to the conclusion that if we cannot have 100% precision (see our previous myth), we simply should not allow matching strategies to act in an automated, unsupervised way on large datasets. While there might be circumstances where this belief is rational, in the context of the scholarly record, this notion is seriously flawed.

First, if you are dealing with any medium to large-sized dataset, it almost certainly contains errors, even before you apply any automated processing to it. Even if data is submitted and curated by users, they can still make mistakes, and might themselves be using automated tools for extracting the data from other sources, without your knowledge. It is thus not entirely obvious that applying an (imperfect) matching strategy to create more relationships would actually make the data quality worse.

Second, while we cannot eliminate all matching errors, we can place a high priority on precision when developing strategies, with the aim of keeping the number of incorrectly matched results as low as possible. We can also make use of additional mechanisms to easily correct for incorrectly matched results, for example doing so manually, in response to error reports.

Finally, the results of matching should always contain provenance information to distinguish them from those that have been manually curated. This way, the users can make their own decisions about whether to use and trust the matching results, relative to their use case.

By applying those additional checks, we can minimise the negative effects of incorrect matching, while at the same time reap the benefits of filling gaps in the scholarly record.

Myth #4: We can only ever guess at the accuracy of our matching results

In attempting to determine the correctness of our matching, we immediately encounter a number of inherent limitations. The sheer amount of entries in many datasets prevents a thorough, manual validation of the results, but if instead, we use too few or specific items as our benchmarks, these are unlikely to be representative of overall performance. The unpredictable nature of future data adds another wrinkle: will our matching always be as successful as when we first benchmarked it or will its performance degrade relative to some change in the data?

With so many unknowns, are we then doomed? No! We have rigorous and scientific tools at our disposal that can help us estimate how accurate our matching will be. How do we use them? Well, that is a big and fairly technical topic, so we will leave you with this little cliffhanger. See you in the next post!

The anatomy of metadata matching

Dominika Tkaczyk — Thu, 27 Jun 2024 00:00:00 +0000

In our previous blog post about metadata matching, we discussed what it is and why we need it (tl;dr: to discover more relationships within the scholarly record). Here, we will describe some basic matching-related terminology and the components of a matching process. We will also pose some typical product questions to consider when developing or integrating matching solutions.

Basic terminology

Metadata matching is a high-level concept, with many different problems falling into this category. Indeed, no matter how much we like to focus on the similarities between different forms of matching, matching affiliation strings to ROR IDs or matching preprints to journal papers are still different in several important ways. At Crossref and ROR, we call these problems matching tasks.

Simply put, a matching task defines the kind or nature of the matching. Examples of matching tasks are bibliographic reference matching, affiliation matching, grant matching, or preprint matching.

Every matching task has an input, which is all the data that is needed to perform the matching. Input data can come in many shapes and forms, depending on the matching task. For example, all of the following could be inputs to a matching task:

Department of Molecular Medicine, Sapporo Medical University, Sapporo 060-8556, Japan

<fr:program xmlns:fr="http://www.crossref.org.turing.library.northwestern.edu/fundref.xsd" name="fundref">
<fr:assertion name="fundgroup">
<fr:assertion name="funder_name">
European Union's Horizon 2020 Research and Innovation Program through Marie Sklodowska Curie
<fr:assertion name="funder_identifier">https://doi-org.turing.library.northwestern.edu/10.13039/501100000780</fr:assertion>
</fr:assertion>
<fr:assertion name="award_number">721624</fr:assertion>
</fr:assertion>
</fr:program>

Everitt, W. N., & Kalf, H. (2007). The Bessel differential equation and the Hankel transform. Journal of Computational and Applied Mathematics, 208(1), 3–19.

{
"title": "Functional single-cell genomics of human cytomegalovirus infection",
"issued": "2021-10-25",
"author": [
{"given": "Marco Y.", "family": "Hein"},
{"given": "Jonathan S.", "family": "Weissman", "ORCID": "http://orcid.org/0000-0003-2445-670X"}
]
}

Every matching task also has an output. For our purposes, this is almost exclusively zero or more matched identifiers. In the context of a specific matching task, output identifiers may be of a specific type (e.g. we might match to a ROR ID, and never to an ORCID ID). In some cases, there can be a certain target set as well (i.e. matching only to DataCite DOIs). The output identifiers can have different cardinality depending on the task, meaning that the matching task might allow for zero, one, or more identifiers as a result of matching to a single input.

A matching strategy defines how the matching is done. Multiple strategies can exist for a specific matching task. Compound strategies can run other strategies and combine their outcomes into a single result.

In some cases, we may also want the matching strategy to output a confidence score for each matched identifier. A confidence score represents the degree of certainty or likelihood that the matched identifier is correct, typically expressed as a value between 0 and 1. This score may help with post-processing or further interpretation of the results.

To summarise, the anatomy of the matching task can be diagrammed as follows:

How to specify a matching task

Whenever we plan the development or integration of a matching solution, it is good to begin by answering a few basic questions:

What problem do we plan to solve with our matching task? What would we call our matching task and how would we describe it?
What do we expect as the input for this matching task? Which input formats do we need to be able to accept? What information do we expect to find in this input?
What kind of identifiers should be output? Is there a target set of identifiers? Can our matching output zero/one/or multiple identifiers, and under what conditions might that occur?

These sound fairly simple, but the answers to these questions can be remarkably complex. Once one tries to apply these concepts to real-world problems, they might encounter several non-obvious challenges.

For example, one common concern is at what level we should define each matching task. Consider the following problems:

Matching bibliographic reference strings to DOIs. Example input:

Everitt, W. N., & Kalf, H. (2007). The Bessel differential equation and the Hankel transform. Journal of Computational and Applied Mathematics, 208(1), 3–19.

Matching structured bibliographic reference to DOIs. Example input:

{
volume: "208",
author: "Everitt",
journal-title: "J. Comput. Appl. Math.",
article-title: "The Bessel differential equation and the Hankel transform",
first-page: "3",
year: "2007",
issue: "1"
}

Are those discrete matching tasks (unstructured reference matching vs. structured reference matching), or are they the same task (reference matching) that can accept different types of inputs (unstructured or structured)?

Similarly, let’s compare the following tasks:

Matching affiliation strings to ROR IDs. Example input:

Department of Molecular Medicine, Sapporo Medical University, Sapporo 060-8556, Japan

Matching funder names to ROR IDs. Example input:

Alexander von Humboldt Foundation

Are these different matching tasks (affiliation matching vs. funder matching), or the same task with different inputs (organisation matching)?

Defining the boundaries of a matching task can also be difficult. Consider, for example, the need to obtain ROR IDs for organisations mentioned in the acknowledgements section of a full-text academic paper. To begin, one may first extract the acknowledgement section from the full text, then run something like a named entity recognition (NER) tool to isolate the organisation names from the extracted text, and finally match these names to ROR IDs. Is this entire process matching, with the input being the full text of a paper? Or perhaps matching starts with the acknowledgement section as the input? Instead, is it only the last phase, where we try to match the extracted name to the ROR ID, that constitutes the matching task, with the extraction phases being completely separate processes?

There are also important questions related to the expected behaviour of a matching strategy. Consider, for example, developing an affiliation matching strategy where we define our input as “an affiliation string”. What should happen when the strategy gets something else on the input, for example, song lyrics? Perhaps the strategy should simply return no matches, or an error, or we could say that in such a situation the behaviour is undefined and it simply doesn’t matter what is returned. But what should happen if in this input we have the lyrics of Street Life by Roxy Music, a song that mentions the names of a few universities that happen to have ROR IDs?

It is likewise important to consider what should happen if different parts of the input match to different identifiers, like in the following example:

Department of Haematology, Eastern Health and Monash University, Box Hill, Australia

Here, “Eastern Health” matches to https://ror.org/00vyyx863 and “Monash University” to https://ror.org/02bfwt286. Should the matching strategy return all the identifiers, one of them (if so, which one?), or nothing at all?

Similar questions arise when it is possible to match to multiple versions (or duplicates) in the target identifier set. This can happen, for example, in the context of bibliographic reference matching or preprint matching. Multiple matches may occur when there are different editions, reprints, or variations of the same publication in the target dataset, each with its own unique identifier.

If you are waiting for an answer to these questions, we unfortunately must disappoint you here. These can only be answered in the context of a specific problem, considering who the users are and what it is they need and expect.

Did you notice any other subtleties related to metadata matching and its concerns? Are there other non-obvious questions that should be considered when planning to develop or integrate metadata matching strategies? Let us know—we’d love to hear from you!

Metadata matching 101: what is it and why do we need it?

Dominika Tkaczyk — Thu, 16 May 2024 00:00:00 +0000

At Crossref and ROR, we develop and run processes that match metadata at scale, creating relationships between millions of entities in the scholarly record. Over the last few years, we’ve spent a lot of time diving into details about metadata matching strategies, evaluation, and integration. It is quite possibly our favourite thing to talk and write about! But sometimes it is good to step back and look at the problem from a wider perspective. In this blog, the first one in a series about metadata matching, we will cover the very basics of matching: what it is, how we do it, and why we devote so much effort to this problem.

What is metadata matching?

Would you be able to find the DOI for the work referenced in this citation?

Everitt, W. N., & Kalf, H. (2007). The Bessel differential equation and the Hankel transform. Journal of Computational and Applied Mathematics, 208(1), 3–19.

We bet you could! You might begin, for example, by pasting the whole citation, or only the title, into a search engine of your choice. This would probably return multiple results, which you would quickly skim. Then you might click on the links for a few of the top results, those that look promising. Some of the websites you visit might contain a DOI. Perhaps you would briefly compare the metadata provided on the website against what you see in the citation. If most of this information matches (see what we did there?), you would conclude that the DOI from that website is, in fact, the DOI for the cited paper.

Well done! You just performed metadata matching, specifically, bibliographic reference matching. Matching in general can be defined as the task or process of finding an identifier for an item based on its structured or unstructured “description” (in this case: finding a DOI of a cited article based on a citation string).

But matching doesn’t have to just be about citations and DOIs. There are many other instances of matching we can think of, for example:

finding the ROR ID for an organisation based on an affiliation string,
finding the ORCID ID for a researcher based on the person’s name and affiliation,
finding the ROR ID for a funder based on the acknowledgements section of a research paper,
finding the grant DOI based on an award number and a funder name.

Matching doesn’t have to be done manually. It is possible to develop fully automated strategies for metadata matching and employ them at scale. It is also possible to use a hybrid approach, where automated strategies assist users by providing suggestions.

Developing automated matching strategies is not a trivial task, and if we want to do it right, it takes a great deal of time and effort. This brings us to our next question: is it worth it?

Why do we need matching?

In short, metadata matching gives us a more complete picture of the research nexus by discovering missing relationships between various entities within and throughout the scholarly record:

These relationships are very powerful. They provide important context for any entity, whether it is a research output, a funder, a research institution, or an author. Imagine for a moment the scholarly record without any such relationships, where all bibliographic references, affiliations (institution names and addresses), and funding information (funder names and grant titles) are provided as unstructured strings only. In such a world, how would you calculate the number of times a particular research paper was cited? How would you get a list of research outputs supported by a specific funder? It would be incredibly challenging to navigate, summarise, and describe research activities, especially considering the scale. Thankfully, these and many other questions can be answered thanks to metadata matching that discovers relationships between entities in the scholarly record.

There are two primary ways we can use metadata matching in our workflows: as semi-automated tools that help users look up the appropriate identifiers or as fully automated processes that enrich the metadata in various scholarly databases.

The first approach is quite similar to the example we described at the beginning. If you are submitting scholarly metadata, for example of a new article to be published, you can use metadata matching to look up identifiers for the various entities and include these identifiers in the submission. For example, with the help of metadata matching, instead of submitting citation strings, you could provide the DOIs for works cited in the paper and instead of the name and address of your organisation, you could provide its ROR ID. To make this easier for people, metadata submission systems and applications sometimes integrate metadata matching tools into user interfaces.

The second approach allows large, existing sources of scholarly metadata to be enriched with identifiers in a fully automated way. For example, we can match affiliation strings to ROR IDs using a combination of machine learning models and ROR’s default matching service, effectively adding more relationships between people and organisations. We can also compare journal articles and preprints metadata in the Crossref database by calculating similarity scores for titles, authors, and years of publication to match them with each other and provide more relationships between preprints and journal articles. This automated enrichment can be done at any point in time, even after research outputs have been formally published.

There are fundamental differences between these two approaches. The first is done under the supervision of a user, and for the second, the matching strategy makes all the decisions autonomously. As a result, the first approach will typically (although not always) result in better quality matches. By contrast, the second approach is much faster, generally less expensive, and scales to even very large data sources.

In the end, no matter what approach is used, the goal is to achieve a more complete accounting of the relationships between entities in the scholarly record.

This blog is the first one in a series about metadata matching. In the coming weeks, we will cover more detail about the product features related to metadata matching, explain why metadata matching is not a trivial problem, and share how we can develop, assess, compare, and choose matching strategies. Stay tuned!

Discovering relationships between preprints and journal articles

Dominika Tkaczyk — Thu, 07 Dec 2023 00:00:00 +0000

In the scholarly communications environment, the evolution of a journal article can be traced by the relationships it has with its preprints. Those preprint–journal article relationships are an important component of the research nexus. Some of those relationships are provided by Crossref members (including publishers, universities, research groups, funders, etc.) when they deposit metadata with Crossref, but we know that a significant number of them are missing. To fill this gap, we developed a new automated strategy for discovering relationships between preprints and journal articles and applied it to all the preprints in the Crossref database. We made the resulting dataset, containing both publisher-asserted and automatically discovered relationships, publicly available for anyone to analyse.

TL;DR

We have developed a new, heuristic-based strategy for matching journal articles to their preprints. It achieved the following results on the evaluation dataset: precision 0.99, recall 0.95, F0.5 0.98. The code is available here.
We applied the strategy to all the preprints in the Crossref database. It discovered 627K preprint–journal article relationships.
We gathered all preprint–journal article relationships deposited by Crossref members, merged them with those discovered by the new strategy, and made everything available as a dataset. There are 642K relationships in the dataset, including:
- 296K provided by the publisher and discovered by the strategy,
- 331K new relationships discovered by the strategy only,
- 15K provided by the publisher only.
In the future, we plan to replace our current matching strategy with the new one and make all discovered relationships available through the Crossref REST API.

Introduction

Relationships between preprints and journal articles link different versions of research outputs and allow one to follow the evolution of a publication over time. The Crossref deposit schema allows Crossref members to provide these relationships for new publications, either as a has-preprint relationship deposited with a journal article, or an is-preprint-of relationship deposited with a preprint.

To assist members who deposit preprints, we also try to connect deposited journal articles with preprints. The current method looks for an exact match between the title and first authors. We send possible matches as suggestions to the preprint server, which decides whether to update the metadata with the relationship.

At the time of writing, 137,837 journal articles in the Crossref database have a has-preprint relationship¹, and 562,225 works of type posted-content (preprints belong to this type) have an is-preprint-of relationship².

We suspected that many preprint–journal article relationships are missing, as some members inevitably fail to deposit them, even after suggestions from the current matching strategy. Another factor is that the current strategy is fairly conservative, and probably misses a significant number of relationships. For these reasons, we decided to investigate whether we could improve on the current process. Doing so would allow us to infer missing relationships on a large scale, similar to how we automatically match bibliographic references to DOIs.

This preprint matching task can be defined in two directions:

We start with a journal article and we want to find all its preprints.
We start with a preprint and we want to find a subsequently published journal article.

On the one hand, matching from journal articles to preprints would allow us to enrich the database continually with new relationships, either periodically or every time new content is added. Since journal articles tend to appear in the database later than their preprints, it makes sense for a new journal article to trigger the matching and not the other way round. This way we can expect the potential matches to be already in the database at the time of matching.

On the other hand, matching from preprints to journal articles can be useful in a situation where we want to add relationships in an existing database retrospectively. In our case, the database contains many more journal articles than preprints, so for performance reasons it is better to start with preprints.

In both cases we are dealing with structured matching, meaning that we match a metadata record of a work (preprint or journal article), rather than unstructured text.

As a result of matching a single preprint or a single journal article, we should expect zero or more matched journal articles/preprints. Multiple matches occur when:

there are multiple versions of the matched preprint and/or
matched works have duplicates.

The image shows the result of matching a journal article to two versions of a preprint:

Matching strategy

Our matching strategy uses the following workflow:

Gathering a short list of candidates using the Crossref REST API.
Scoring the similarity between the input item and each candidate.
A final decision about which candidates, if any, should be returned as matches.

Gathering candidates is done using the Crossref REST API’s query.bibliographic parameter. The query is a concatenation of the title and authors’ last names of the input item. We filter the candidates based on their type, to leave only preprints or only journal articles, depending on the direction of the matching. In the future, instead of getting the candidates from the REST API, we will be using a dedicated search engine, optimised for preprint matching.

Scoring candidates is heuristic-based. Similarities between titles, authors, and years are scored independently, and the final score is their average. Titles are compared in a fuzzy way using the rapidfuzz library. Authors are compared pairwise using the ORCID ID, or first/last names if ORCID ID is not available. The similarity score between issued years is 1 if the article was published no earlier than one year before the preprint and no later than three years after the preprint, or 0 otherwise.

The final decision is made based on two parameters: minimum score and maximum score difference, both chosen based on a validation dataset. The following diagram depicts the results of applying these two parameters in all possible scenarios. First, any candidate scoring below the minimum score is rejected (grey area in the diagram). Second, the scores of the remaining candidates are compared with the score of the top candidate. If the score of a candidate is close enough to the score of the top candidate, it is returned as a match (blue area).

This process can result in the following scenarios:

Scenario A: there is no candidate above the minimum score. This means nothing matches sufficiently, so nothing is returned.
Scenario B: there is only one candidate above the minimum score. This means it is the best match and we don’t have much of a choice, so it is returned.
Scenario C: there are multiple candidates above the minimum score, and they all have similar scores. This means they all are similarly good matches, so all are returned.
Scenario D: there are multiple candidates above the minimum score, but their scores differ a lot. In this case, we don’t want to return all of them, but only those that are close to the top match. Intuitively, we don’t want to return less-than-great matches if we have really great ones. This is when the maximum score difference comes into play: we return the candidates with the “score distance” to the top candidate lower than the maximum score difference.

We evaluated this strategy on a test set sampled from the Crossref metadata records. The test set contains 3,000 pairs (journal article, set of corresponding preprints). Half of the journal articles have known preprints and the other half don’t. The test set can be accessed here.

We used precision, recall, and F0.5 as evaluation metrics:

Precision measures the fraction of the matched relationships that are correct.
Recall measures the fraction of the true relationships that were matched.
F0.5 combines precision and recall in a way that favours precision.

The strategy achieved the following results: precision 0.9921, recall 0.9474, F0.5 0.9828. The average processing time was 0.96s.

We have made this strategy (journal article -> preprints) available through the (experimental) API: https://marple-research-crossref-org.turing.library.northwestern.edu/match?task=preprint-matching&strategy=preprint-sbmv&input=10.1109/access.2022.3213707. The input is the DOI of a journal article we want to match to preprints, and the output is a list of matches found, along with the score for each.

We have investigated other approaches to making decisions about which candidates to return as matches (step 3 above), including using machine learning. At present none have outperformed the heuristic approach described above. The heuristic method is also preferred because of its fast performance.

Preprint–journal article relationship dataset

We applied the strategy to the entire Crossref database:

We selected all preprints published until the end of August 2023. This included only works with type posted-content and subtype preprint, as reported by the REST API. There were 1,050,247 of them.
We ran the matching strategy (preprint -> journal article) on them. This resulted in 627,011 preprint–journal article relationships.
The resulting relationships were combined with the relationships deposited by the Crossref members. We included relationships of types has-preprint or is-preprint-of, where both sides of the relationship exist in our database, were published until the end of August 2023, and are of proper types and subtypes (type=journal-article for the journal article and type=posted-content, subtype=preprint for the preprint).

The resulting dataset is a single CSV file with the following fields:

preprint DOI (string)
journal article DOI (string)
whether the publisher of the journal article deposited this relationship (boolean)
whether the publisher of the preprint deposited this relationship (boolean)
the confidence score returned by the strategy (float, empty if the strategy did not discover this relationship)

The dataset contains:

641,950 relationships in total, including 580,532 preprints and 565,129 journal articles,
14,939 of them were deposited by the Crossref members, but not discovered by the strategy,
330,826 of them were discovered by the strategy, but not provided by any Crossref member,
296,185 of them were both deposited by a Crossref member and discovered by the strategy.

The dataset can be downloaded here.

Conclusions and what’s next

Overall, based on the number of existing and newly discovered preprint–journal article relationships, it seems that employing automated matching strategies would approximately double the number of these relationships in the Crossref database. In the future, we would like to match new journal articles on an ongoing basis. We also plan to make all discovered relationships available through the REST API.

In the meantime, we will be publishing the discovered relationships in the form of datasets, and we invite anyone interested to further analyse this data. And if you find out something interesting about preprints and their relationships, do let us know!

The more the merrier, or how more registered grants means more relationships with outputs

Dominika Tkaczyk — Wed, 22 Feb 2023 00:00:00 +0000

One of the main motivators for funders registering grants with Crossref is to simplify the process of research reporting with more automatic matching of research outputs to specific awards. In March 2022, we developed a simple approach for linking grants to research outputs and analysed how many such relationships could be established. In January 2023, we repeated this analysis to see how the situation changed within ten months. Interested? Read on!

TL;DR

The overall numbers changed a lot between March 2022 and January 2023:
- the total number of registered grants doubled (from ~38k to ~76k)
- the total numbers of relationships established between grants and research outputs quadrupled (from 21k to 92k)
- the percentage of linked grants increased substantially (from 10% to 23%)
Most of this growth can be attributed to one funder, the European Union. They started registering grants with us in December 2022, and:
- their grants constitute 47% of all grants registered by January 2023 and 95% of grants registered between March 2022 and January 2023
- 72% of all established relationships involve their grants
We have further work planned both internally and with the community to consolidate and build out important relationships between funding and research outputs.

Introduction

When we started to develop, think and talk about grant registration at Crossref back in 2017, one of the key things we expected this to support was easier, more efficient, accurate analysis of research outputs funded by specific awards.

This is backed up by conversations with funders who are keen to fill in gaps in the map of the research landscape with new data points and better quality information, search for grants, investigators, projects or organisations associated with awards and simplify the process of research reporting and with automatic matching of outputs to grants.

This is in keeping with and informed our recent recommendations about how funding agencies can meet open science guidance using existing open infrastructure, which included input from ORCID and DataCite. It’s also in keeping with recent studies on how important funding and grant metadata is to help the community use this information in their own research.

To meet these expectations, we need not only identifiers and metadata of grants, but also relationships between them and research outputs supported by them. Unfortunately, our schema does not make it easy to directly deposit such relationships, and so there are only a handful of them available. But we wouldn’t let such a minor obstacle stop us! In March 2022 we analysed the metadata of registered grants and developed a simple matching approach to automatically link grants to research outputs supported by them. Back then, we were able to find 20,834 relationships, involving 17,082 research outputs and 3,858 grants (which was 10% of all registered grants).

Now that we are seeing the accumulation of grant metadata being registered with Crossref, we have a bigger dataset to test these expectations against than we did a year ago. So we decided to do the analysis again. And the results are in, they’re open, and they’re positive. We’ll explain below.

The methodology

To spare you from having to read the old analysis in detail, here is a very brief summary of the matching methodology. To find relationships between grants and research outputs, we iterated over all registered grants, and for each grant we searched for research outputs that looked like they might have been supported by this grant. We established a relationship between a grant and a research output if one of the following three scenarios was true:

The research output contained the DOI of the grant (deposited as the award number).
The award number in the grant was the same as the award number in the research output, the research output contained the funder ID, and one of the following was true:
a. Funder ID in the grant was the same as the funder ID in the research output
b. Funder ID in the grant replaced or was replaced by the funder ID in the research output
c. Funder ID in the grant was an ancestor or the descendant of the funder ID in the research output
The award number in the grant was the same as the award number in the research output, the research output did not contain the funder ID, and one of the following was true:
a. Funder name in the research output was the same as the funder name in the grant
b. Funder name in the research output was the same as the name of a funder that replaced or was replaced by the funder in the grant
c. Funder name in the research output was the same as the name of an ancestor or a descendant of the funder in the grant

Note that the replaced/replaced-by relationships and ancestor/descendant hierarchy are taken from the Funder Registry.

Current results

Since March 2022, six additional funders have started registering grants with us. As a result, the total number of grants doubled, and the total number of established relationships between grants and research outputs, linked grants, and linked research outputs quadrupled. Here is the comparison of the total numbers of grants, established relationships, linked grants, and linked research outputs in March 2022 and in January 2023:

95% of grants registered within ten months between March 2022 and January 2023 were registered by one funder: the European Union. This suggests that this funder contributed a lot to this rapid increase in the number of established relationships. It looks like this funder’s grant metadata is of high quality and matches well the funding information given in the research outputs supported by this funder’s grants.

Let’s also compare the breakdowns of all established relationships by the matching method:

The distributions are a bit different. Currently, the percentage of relationships established based on the replaced/replaced-by relationship is much smaller than before, suggesting that newer data uses correct funder IDs instead of deprecated ones. Also, the percentage of the relationships matched by the funder ID increased from 40% to 48%, which is great, because this is the most reliable way of matching.

And here we have the statistics broken down by grant registrants. Only funders with at least 100 registered grants are included. The table shows the number of relationships, grants, linked grants, and linked research outputs, and is sorted by the percentage of linked grants.

funder	relationships	linked research outputs	grants	linked grants
European Union	66,562	60,630	35,530	12,688 (36%)
Gordon and Betty Moore Foundation	93	92	113	33 (29%)
Japan Science and Technology Agency (JST)	15,584	13,464	9,923	2,323 (23%)
James S. McDonnell Foundation	519	513	577	121 (21%)
Melanoma Research Alliance	188	185	425	82 (19%)
Muscular Dystrophy Association	50	50	178	25 (14%)
Parkinson’s Foundation	30	29	107	15 (14%)
Asia-Pacific Network for Global Change Research	127	127	560	70 (13%)
The ALS Association	96	90	477	58 (12%)
Wellcome	8,868	6,436	17,537	1,735 (10%)
American Cancer Society	19	19	266	15 (6%)
Templeton World Charity organisation	2	2	281	2 (0.7%)
Office of Scientific and Technical Information (OSTI)	73	69	8,723	62 (0.7%)
Children’s Tumor Foundation	1	1	662	1 (0.1%)

There are substantial differences between the percentages of linked grants from different funders. One of the newest registrants, the European Union, is at the top of the table with 36% of their grants linked to research outputs. This further confirms the high quality of the metadata registered by this member. It is worth noticing that this member is responsible for the majority of the growth reported here as they cover Horizon Europe, the European Research Council, and many other funding bodies and schemes.

Why are these percentages so low for some funders? It could be caused by systematic discrepancies between the award numbers attached to the grants and those reported in research outputs. It could also be the case that most grants registered by a given funder are new grants, and the research outputs supported by them simply have not been published yet. Time will tell!

What’s next

We’re dedicating lots of time in 2023 to examine, evolve, and expose the matching we do and can do at Crossref across different metadata fields. We then plan to incorporate matching improvements into our services so that everyone can benefit.

This isn’t a standalone piece of work. As you can see, the more award metadata we have connected to grants by funders and connected to outputs by those who post or publish research, the better we’ll be able to do this. To make it easier for more funders to participate, and based on funder feedback, we’ve built a simple tool for members to register their grants. We will also work to help incorporate grant identifiers into publishing and funder workflows, and further our discussions with the funders in our Funder Advisory Group and the wider community, including working together with the Open Research Funders Group, the HRA, Altum, Europe PMC, the OSTP, and the ORCID Funder Interest Group. And there will be more to come as we work together to consolidate and build out important relationships between funding and outputs - for everyone.

Follow-up

Every new thing takes time to get off the ground and to show evidence of its value. We’ve seen a significant step forward recently with funders joining and contributing to the research nexus. Publishers have been contributing funding data for years, and it’s now becoming much clearer to see how these two communities and these two sets of metadata are coming together to make research smoother and easier to manage and evaluate. If you are ready to register grants, talk about linking up your outputs, or just want to learn more about this work, we’d love to hear from you.

Follow the money, or how to link grants to research outputs

Dominika Tkaczyk — Tue, 22 Mar 2022 00:00:00 +0000

The ecosystem of scholarly metadata is filled with relationships between items of various types: a person authored a paper, a paper cites a book, a funder funded research. Those relationships are absolutely essential: an item without them is missing the most basic context about its structure, origin, and impact. No wonder that finding and exposing such relationships is considered very important by virtually all parties involved. Probably the most famous instance of this problem is finding citation links between research outputs. Lately, another instance has been drawing more and more attention: linking research outputs with grants used as their funding source. How can this be done and how many such links can we observe?

TL;DR

We looked for links between research outputs and grants registered with Crossref.
Grant DOIs alone are not enough for linking research outputs with grants, because the funding information in research outputs typically does not contain grant DOIs (yet). Award numbers alone are also not enough because they are not globally unique.
We used either grant DOIs (if available) or the combination of award number and funder information to match grants to research outputs.
In total, we found 20,834 links between research outputs and registered grants, involving 17,082 research outputs and 3,858 grants (10% of all registered grants)¹.
Erroneous and incomplete metadata, especially involving award numbers, is the main factor that prevents linking research outputs to grants.

Introduction

The ecosystem of scholarly metadata is filled with relationships between items of various types: a person authored a paper, an author works at a university, a paper cites a book, a book contains a chapter, a funder funded research. Those relationships are absolutely essential: an item without them is missing the most basic context about its structure, origin, and impact.

No wonder that finding and exposing relationships between items in the scientific ecosystem is considered very important by virtually all parties involved. Probably the most famous instance of this problem is finding citation links between research outputs. Another, relatively new example, is linking research outputs with grants used as their funding source.

At Crossref, for some time now we have been seeing a steady growth of funder membership and grant registration. We are aware that the possibility of finding relationships between grants and research outputs is a big reason why funders are registering grants with us in the first place. Being able to see which research outputs are being supported by which grants helps reduce the reporting burden on researchers, funders, and institutions alike, especially now with the addition of ROR IDs to help complete the picture. Exposing relationships between research outputs and grants also increases the transparency of funding sources of the research, making it easier to assess and trust scientific findings.

But how can we find those relationships and how many of them can we already observe? Thankfully our REST API, recently equipped with the grant metadata, can help us answer these questions.

The perfect scenario

Imagine a world where the metadata of any scientific output states all relationships with other items existing in the scientific ecosystem, and those related items are always referred to by their persistent identifiers, allowing all this information to be accessed in a fully machine-readable way… Lovely, right?

In the case of citations, in such a perfect world every bibliographic reference has a DOI of the cited item. And in the case of funding information, a scientific paper contains grant DOIs, stating the funded-by relationships between the paper and the grants.

But, as the last two years have painfully taught us all, life is not all rainbows and unicorns.

The reality kicks in

We know that around 71% of bibliographic references are deposited with Crossref without a DOI of the cited item. This means that if we want to establish citation links between items, we need to match the bibliographic references using the provided metadata, which is not a trivial task and can potentially introduce errors.

And the situation with the funding information and grant DOIs is even worse.

Problem #1: our schema does not allow the publishers to attach grant DOIs to research outputs

This issue is 100% on us. Because grant DOIs are relatively new, our deposit schema does not yet allow to specify the grant DOI in the funding information of a research output, even if the publisher wanted to. We are working on changing this.

Interestingly, it looks like persistent identifiers always find a way. Within over 7.4 million research outputs with funding information, we noticed 6 cases where a grant DOI was provided as an award number. For example in 10.1093/nar/gkaa994 we have the following:

funder: [
{
name: "Wellcome Trust",
award: ["10.35802/108758"],
doi-asserted-by: "publisher",
DOI: "10.13039/100010269"
}, ...
]

This may not be 100% correct from the schema perspective, but it is very useful when one is interested in linking grants to research outputs!

But those cases are extremely rare outliers. For the vast majority of the outputs, grant DOIs are not present in the metadata. This means that, just like in the case of bibliographic references, we have to use the metadata to match funding information to grants.

Funding information is typically given as a pair: award number, funder information. Grants contain similar metadata. One might be tempted to use only the award number for linking, as in some cases it can look like a grant identifier.

Let’s consider an example. We want to find all papers funded by grant 10.37807/gbmf7622. The award number is GBMF7622. A simple approach might be to search for items with this award number in Crossref’s REST API, which returns 12 results². However, one of the resulting items is the grant itself³. So excluding that, it seems like there are 12-1=11 research outputs funded by this grant.

Simple and easy, right? Well, think again.

Problem #2: award numbers are not unique

Let’s look at another example grant: 10.25585/60000600. Its award number is 2817 and the funder is the US Department of Energy.

When we search for this award we get 10 results⁴. Like before, one of them is our grant. After examining the remaining 9 we will see that:

3 items have been funded by the Joint Genome Institute, which according to the Funder Registry has been incorporated into Basic Energy Sciences, which is a descendant of the US Department of Energy
2 items have been funded by International Rett Syndrome Foundation from the US
2 items have been funded by Agencia Nacional de Promoción Científica y Tecnológica from Argentina
1 item has been funded by Arak University of Medical Sciences from Iran
1 item has been funded by Shahrekord University also from Iran

So among only 9 items mentioning the same award number we have in fact 5 different grants. Our input grant should probably be linked only to the three items mentioning Joint Genome Institute. The main problem illustrated here is that the award numbers are not globally unique, and thus should not be treated like identifiers.

Indeed, within 38,326 grants registered so far, we have 37,608 distinct award numbers, and among those, there are 716 award numbers, each of which appears in multiple grants. This issue comes in two flavours: conflicts between and within funders.

Between-funder award number conflicts

A conflict between funders is when more than one funder uses the same award number for one of their grants. This is expected - award numbers are assigned by funders internally and are not designed to be a globally unique identifier.

Out of 716 award numbers that appear in multiple grants, 12 are numbers that appear in grants of different funders. For example, there are two grants with the award number 105626:

Systemic MFG-E8 Blockade as Melanoma Therapy funded by Melanoma Research Alliance
Institutional Strategic Support Fund Phase2 FY2014/16 funded by Wellcome Trust

Because of those conflicts, we cannot simply rely on the award numbers for linking grants to research outputs. Instead, we have to use more information to be sure that the links are correctly established.

Within-funder award number conflicts

To our big surprise, it turns out that the majority of the award number conflicts happen not between different funders, but within the grants of a single funder. Out of 716 award numbers that appear in multiple grants, 704 appear in multiple grants of a single funder only. Such situations are not expected and could indicate an error or some other systematic issue with the data.

Interestingly, out of those 704 award numbers, 700 are associated with the US Department of Energy. We’ve followed up with them in order to clarify or resolve this. The US Department of Energy pointed out a fundamental issue with the data model: currently a grant deposited with Crossref has to have at least one funder DOI, and no other way of identifying the associated organisation is allowed. At the same time, some of the facilities that should appear in their grants’ metadata are not funders at all and thus cannot be identified by a funder DOI. In the future, they plan to identify those facilities in their grant metadata by providing ROR IDs.

Because of within-funder award number conflicts, in some cases it might be difficult to distinguish between two grants with the same award number and funder. A solution might be to use additional information or simply not accept any links if a research output cannot be reliably linked to one grant only.

Our linking approach

Based on all those observations, we adopted the following approach:

We iterated over all registered grants, for each we performed the following steps:
- We used award.number:<grant DOI> filter in the REST API to find all items listing a given grant’s DOI as the award number. Because this is based on the grant’s persistent identifier, we recorded those links without any further verification.
- We used the award.number:<grant award number> filter in the REST API to find all items listing grant’s award number in the funding information. Each resulting item was then verified by comparing the funder information in the item to the funder information in the grant. We recorded the link between the grant and the candidate item only if the verification succeeded.
In the final step, we examined all recorded links to make sure that each pair (research output, award number) is linked to at most one grant. Links violating this rule were flagged as not reliable.

We used different techniques to verify the funder information between the research output (item) and the grant, depending on what information is available. Grants always have the funder DOI. The item, however, can have the funder DOI, the funder name, or both.

If the funder DOI was available on both sides, the following rules were used for the funder verification (ordered by decreasing confidence):

Both the item and the grant contain the same funder DOI, for example, 10.35802/089928 and 10.1242/jcs.196758
The funder in the item replaced or was replaced by the funder in the grant (according to the Funder Registry), for example, 10.35802/104848 and 10.1136/medethics-2020-106821
The funder in the paper is an ancestor or a descendant of the funder in the grant (according to the Funder Registry), for example, 10.46936/sthm.proj.2010.40084/60004575 and 10.1016/j.heliyon.2018.e00629

If the funder DOI was not available in the item, the following rules were used for the funder verification (ordered by decreasing confidence):

The funder name in the paper is the same (ignoring the case) as the funder name in the grant, for example, 10.35802/110166 and 10.12688/wellcomeopenres.14645.4
The funder name in the item is the same (ignoring the case) as the name of the funder that replaced/was replaced by the funder in the grant, for example, 10.35802/206194 and 10.1172/jci.insight.96381
The funder name in the item is the same (ignoring the case) as the name of the ancestor/descendant of the funder in the grant, for example, 10.46936/cpbl.proj.2001.2191/60002922 and 10.1109/tkde.2016.2628180

Note that this is in fact very similar to our reference matching approach. In both cases, first we search for candidate items, and then verify the candidates by comparing the metadata. The actual metadata used for the verification varies, because different information is typically given in the bibliographic reference and the funding information.

What we found

This procedure applied to the entire Crossref dataset resulted in 20,846 links between research outputs and grants⁵. Of those, 12 were flagged as unreliable, because they involved more than one grant linked to the same item and award number. The rest of this section focuses on the remaining 20,834 links.

Within the 20,834 links, we have 17,082 research outputs and 3,858 (10.1%) grants.

Here is the breakdown into the verification approaches used:

Verification	#links	%links
The item contains grant DOI - no verification	6	<0.1%
Funder DOIs are the same	8,364	40.1%
Funder DOIs are related with a replaced/was replaced by relationship	3,704	17.8%
Funder DOIs are related with an ancestor/descendant relationship	7,718	37.0%
Funder names are the same	591	2.8%
The name of the funder in the item is the same as the name of the funder that replaced/was replaced by the funder in the grant	364	1.7%
The name of the funder in the item is the same as the name of the ancestor or descendant of the funder in the grant	87	0.4%

In most cases, just using the funder DOIs for the verification was enough. Verifying by the funder name added 1,042 links, which is 5% of all links.

And here are statistics for individual funders. Only funders with at least 10 deposited grants are listed in the table. The table shows the number of detected links, the number of distinct research outputs linked, the total number of outputs mentioning the given funder DOI, and the number of grants.

Funder	#links	#linked research outputs	#total outputs with funder DOI	#grants
Japan Science and Technology Agency	11,922	10,411	25,779	9,383
Wellcome Trust (including both funder DOIs 10.13039/100004440 and 10.13039/100010269)	8,001	6,246	49,492	17,534
James S. McDonnell Foundation	463	457	2,534	557
Melanoma Research Alliance	152	150	894	392
Asia-Pacific Network for Global Change Research	100	100	838	539
ALS Association	84	78	909	434
U.S. Department of Energy	56	52	97,482	8,462
Gordon and Betty Moore Foundation	51	50	5,928	94
American Cancer Society	3	3	7,276	107
Children’s Tumor Foundation	1	1	759	630
American Parkinson Disease Association	0	0	181	12
Neurofibromatosis Therapeutic Acceleration Program	0	0	101	68
International Anesthesia Research Society	0	0	94	34
Australian National Data Service	0	0	92	67

Note that the fourth column reports the total number of outputs registered with Crossref and mentioning the given funder DOI, including grants, journal papers and all other record types.

It is interesting to compare the number of linked research outputs for a given funder with the total number of research outputs mentioning a given funder DOI. In general, for a funder that registers grants, the more research outputs mentioning this funder, the more links we should be able to find.

And for some funders (Japan Science and Technology Agency, Melanoma Research Alliance, Asia-Pacific Network for Global Change Research, Wellcome Trust, James S. McDonnell Foundation), the number of linked outputs is indeed high, as compared with how many outputs mention the funder in the first place. This suggests our procedure was quite successful in linking outputs funded by these funders, meaning that in general the metadata in their grants and the funding information in the research outputs match.

On the other hand, we have a few funders for which we managed to link only a very small fraction of research outputs. There are several potential explanations here. A simple one is that not all relevant grants have been deposited yet. For example, a funder might be registering new grants only, whereas many research outputs mention older, not yet registered grants. It is also possible that there are systematic differences in how the publishers deposit the funding information in articles and other outputs, and how it is given in grants. Such differences might prevent us from establishing links, contributing to the overall low percentage of linked grants.

The importance of being precise

Here are some examples of existing links that should’ve been found, but were not.

The award number in grant 10.48105/pc.gr.93156 is CTF-2020-01-004. This article: 10.3390/ijms22094716 mentions award number 2020‐01‐004 and the same funder (Children’s Tumor Foundation). It is very probable that this is the same grant, but our procedure expects exactly the same award number, and so the two were not linked.

Paper 10.1128/genomea.00159-18 contains award number 1931 and U.S. Department of Energy as the funder. There are two grants with the same award number and funder: 10.46936/10.25585/60001053 and 10.46936/genr.proj.2000.1931/60002530. It is difficult to choose between them, and these links were marked as unreliable.

These examples could be signs of systematic errors and/or discrepancies that effectively prevent linking of those funders’ grants.

What’s next

In problems such as linking grants to research outputs, there are typically two key ingredients of the success, which at the same time are the main areas of improvement: the quality of the metadata, and the strength of the linking approach.

The metadata could be improved greatly by addressing existing discrepancies between grants and research outputs and allowing (and encouraging!) the publishers to provide grant DOIs in the funding information. Thankfully, we are not alone in those efforts. Both this recent Upstream blog from Alexis-Michel Mugabushaka, and this Scholarly Kitchen post from Robert Harrington call for the development and adoption of grant DOIs in scholarly metadata.

In terms of the linking approach, there are some ideas that could be used to further improve the linking accuracy and completeness:

The verification by funder name could be fuzzy and allow for minor variations like typos or additional words.
Apart from replaced/replaced by and ancestor/descendant, there are other relationships between funders in the Funder Registry: continuation of, incorporates/incorporated into, merged with, renamed as, split into/split from. We could also consider those relationships during the funder validation.
Apart from the funder information, there is other information that could be potentially used for verification, for example, the names of the authors and the investigators, the domain, or keywords.

If you have any questions, do get in touch!

All numbers are as of March 8, 2022 ↩︎
https://api-crossref-org.turing.library.northwestern.edu/works?filter=award.number:gbmf7622 ↩︎
https://api-crossref-org.turing.library.northwestern.edu/works?filter=award.number:gbmf7622,type:grant ↩︎
https://api-crossref-org.turing.library.northwestern.edu/works?filter=award.number:2817 ↩︎
The code and data available here: https://gitlab.com/crossref/labs_data_analyses/-/tree/master/analyses/22-01-26-grants-matching ↩︎

What if I told you that bibliographic references can be structured?

Dominika Tkaczyk — Mon, 08 Jul 2019 00:00:00 +0000

Last year I spent several weeks studying how to automatically match unstructured references to DOIs (you can read about these experiments in my previous blog posts). But what about references that are not in the form of an unstructured string, but rather a structured collection of metadata fields? Are we matching them, and how? Let’s find out.

TL;DR

43% of open/limited references deposited with Crossref have no publisher-asserted DOI and no unstructured string. This means they need a matching approach suitable for structured references. [EDIT 6th June 2022 - all references are now open by default].
I adapted our new matching algorithms: Search-Based Matching (SBM) and Search-Based Matching with Validation (SMBV) to work with both structured and unstructured references.
I compared three matching algorithms: Crossref’s current (legacy) algorithm, SBM and SBMV, using a dataset of 2,000 structured references randomly chosen from Crossref’s references.
SBMV and the legacy algorithm performed almost the same. SBMV’s F1 was slightly better (0.9660 vs. 0.9593).
Similarly as in the case of unstructured references, SBMV achieved slightly lower precision and better recall than the legacy algorithm.

Introduction

Those of you who often read scientific papers are probably used to bibliographic references in the form of unstructured strings, as they appear in the bibliography, for example:

[5] Elizabeth Lundberg, “Humanism on Gallifrey,” Science Fiction Studies, vol. 40, no. 2, p. 382, 2013.

This form, however, is not the only way we can store the information about the referenced paper. An alternative is a structured, more machine-readable form, for example using BibTeX format:

@article{Elizabeth_Lundberg_2013,
year = 2013,
publisher = {{SF}-{TH}, Inc.},
volume = {40},
number = {2},
pages = {382},
author = {Elizabeth Lundberg},
title = {Humanism on Gallifrey},
journal = {Science Fiction Studies}
}

Probably the most concise way to provide the information about the referenced document is to use its identifier, for example (🥁drum roll…) the DOI:

<https://doi-org.turing.library.northwestern.edu/10.5621/sciefictstud.40.2.0382>

It is important to understand that these three representations (DOI, structured reference and unstructured reference) are not equivalent. The amount of information they carry varies:

The DOI, by definition, provides the full information about the referenced document, because it identifies it without a doubt. Even though the metadata and content are not directly present in the DOI string, they can be easily and deterministically accessed. It is by far the preferred representation of the referenced document.
The structured reference contains the metadata of the referenced object, but it doesn’t identify the referenced object without a doubt. In our example, we know that the paper was published in 2013 by Elizabeth Lundberg, but we might not know exactly which paper it is, especially if there are more than one document with the same or similar metadata.
The unstructured reference contains the metadata field values, but without the names of the fields. This also doesn’t identify the referenced document, and even its metadata is not known without a doubt. In our example, we know that the word “Science” appears somewhere in the metadata, but we don’t know for sure whether it is a part of the title, journal title, or maybe the author’s (very cool) name.

The diagram presents the relationships between all these three forms:

The arrows show actions that Crossref has to perform to transform one form to another.

Green transformations are in general easy and can be done without introducing any errors. The reason is that green arrows go from more information to less information. We all know how easy it is to forget important stuff!

Green transformations are typically performed when the publication is being created. At the beginning the author can access the DOI of the referenced document, because they know exactly which document it is. Then, they can extract the bibliographic metadata (the structured form) of the document based on the DOI, for example by following the DOI to the document’s webpage or retrieving the metadata from Crossref’s REST API. Finally, the structured form can be formatted into an unstructured string using, for example, the CiteProc tool.

We’ve also automated it further and these two green transformation (getting the document’s metadata based on the DOI and formatting it into a string) can be done in one go using Crossref’s content negotiation.

Red transformations are often done in systems that store bibliographic metadata (like our own metadata collection), often at a large scale. In these systems, we typically want to have DOIs (or other unique identifiers) of the referenced documents, but in practise we often have only structured and/or unstructured form. To fix this, we match references. Some systems also perform reference parsing (thankfully, we discovered we do not need to do this in our case).

In general, red transformations are difficult, because we have to go from less information to more information, effectively recreating the information that has been lost during paper writing. This requires a bit of reasoning, educated guessing, and juggling probabilities. Data errors, noise, and sparsity make the situation even more dire. As a result, we do not expect any matching or parsing algorithm to be always correct. Instead, we perform evaluations (like in this blog post) to capture how well they perform on average.

My previous blog post focused on matching unstructured references to DOIs (long red “matching” arrow). In this one, I analyse how well we can match structured references to DOIs (short red “matching” arrow).

References in Crossref

You might be asking yourself how important it is to have the matching algorithm working for both structured and unstructured references. Let’s look more closely at the references our matching algorithm has to deal with.

29% of open/limited references deposited with Crossref already have the DOI provided by the publisher member. At Crossref, when we come across those references, we start dancing on a rainbow to the tunes of Linkin Park, while the references holding their DOIs sprinkle from the sky. Some of us sing along. We live for those moments, so if you care about us, please provide as many DOIs in your references as possible!

You might be wondering how we are sure these publisher-provided DOIs are correct. The short answer is that we are not. After all, the publisher might have used an automated matcher to insert the DOIs before depositing the metadata. Nevertheless, our current workflow assumes these publisher-provided DOIs are correct and we simply accept them as they are.

Unfortunately, the remaining 71% of references are deposited without a DOI. Those are the references we try to match ourselves.

Here is the distribution of all the open/limited references:

17% of the references are deposited with no DOI and both structured and unstructured form. 11% have no DOI and only an unstructured form, and 43% have no DOI and only a structured form. These 43% cannot be directly processed by the unstructured matching algorithm.

This distribution clearly shows that we need a matching algorithm able to process both structured and unstructured references. If our algorithm worked only with one type, we would miss a large percentage of the input references, and the quality of our citation metadata would be questionable.

The analysis

Let’s get to the point. I evaluated and compared three matching algorithms, focusing on the structured references.

The first algorithm is one of the legacy algorithms currently used in Crossref. It uses fuzzy querying in a relational database to find the best matching DOI for the given structured reference. It can be accessed through a Crossref OpenURL query.

The second algorithm is an adaptation of the Search-Based Matching (SBM) algorithm for structured references. In this algorithm, we concatenate all metadata fields of the reference and use it to search in the Crossref’s REST API. The first hit is returned as the target DOI if its relevance score exceeds the predefined threshold.

The third algorithm is an adaptation of the Search-Based Matching with Validation (SBMV) for structured references. Similarly as in the case of SBM, we also concatenate all metadata fields of the input reference and use it to search in the Crossref’s REST API. Next, a number of top hits are considered as candidates and their similarity score with the input reference is calculated. The candidate with the highest similarity score is returned as the target DOI if its score exceeds the predefined threshold. The similarity score is based on fuzzy comparison of the metadata field values between the candidate and the input reference.

I compared these three algorithms on a test set composed of 2,000 structured bibliographic references randomly chosen from Crossref’s metadata. For each reference, I manually checked the output of all matching algorithms, and in some cases performed additional manual searching. This resulted in the true target DOI (or null) assigned to each reference.

The metrics are the same as in the previous evaluations: precision, recall and F1 calculated over the set of input references.

The thresholds for SBM and SBMV algorithms were chosen on a separate validation dataset. The validation dataset also contains 2,000 structured references with manually-verified target DOIs.

The results

The plot shows the results of the evaluation of all three algorithms:

The vertical black lines on top of the bars represent the confidence intervals.

As we can see, SBMV and the legacy approach achieved very similar results. SBMV slightly outperforms the legacy approach in F1: 0.9660 vs. 0.9593.

SBMV is slightly worse that the legacy approach in precision (0.9831 vs. 0.9929) and better in recall (0.9495 vs. 0.9280).

The SBM algorithm performs the worst, especially in precision. Why is there such a huge difference between SBM and SBMV? The algorithms differ in the post-processing validation stage. SBM relies on the ability of the search engine to select the best target DOI, while SBMV re-scores a number of candidates obtained from the search engine using custom similarity. The results here suggest that in the case of structured references, the right target DOI is usually somewhere close to the top of the search results, but often it is not in the first position. One of the reasons might be missing titles in 76% of the structured references, which can confuse the search engine.

Let’s look more closely at a few interesting cases in our test set:

first-page = 1000
article-title = Sequence capture using PCR-generated probes: a cost-effective method of targeted high-throughput sequencing for nonmodel organisms
volume = 14
author = Peñalba
year = 2014
journal-title = Molecular Ecology Resources

The reference above was successfully matched by SBMV to https://doi-org.turing.library.northwestern.edu/10.1111/1755-0998.12249, even though the document’s volume and pages are missing from Crossref’s metadata.

issue = 2
first-page = 101
volume = 6
author = Abraham
year = 1987
journal-title = Promoter: An Automated Promotion Evaluation System

Here the structure incorrectly labels article title as journal title. Despite this, the reference was correctly matched by our brave SBMV to https://doi-org.turing.library.northwestern.edu/10.1287/mksc.6.2.101.

author = Marshall Day C.
volume = 39
first-page = 572
year = 1949
journal-title = India. J. A. D. A.

Above we have most likely a parsing error. A part of the article title appears in the journal name, and the main journal name is abbreviated. ‘I see what you did there, my old friend Parsing Algorithm! Only a minor obstacle!’ said SBMV, and matched the reference to https://doi-org.turing.library.northwestern.edu/10.14219/jada.archive.1949.0114.

volume = 5
year = 2015
article-title = A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels
journal-title = BMJ Open

Here the the page number and author are not in the structure, but our invincible SBMV jumped over the holes left by the missing metadata and gracefully grabbed the right DOI https://doi-org.turing.library.northwestern.edu/10.1136/bmjopen-2015-009138.

issue = 2
first-page = 533
volume = 30
author = Uthman BM
year = 1989
journal-title = Epilepsia

In this case we have a mismatch in the page number (“533” vs. “S33”). But did SBMV give up and burst into tears? I think we already know the answer! Of course, it conquered the nasty typo with the sword made of fuzzy comparisons (yes, it’s a thing!) and brought us back the correct DOI https://doi-org.turing.library.northwestern.edu/10.1111/j.1528-1157.1989.tb05823.x.

Structured vs. unstructured

How does matching structured references compare to matching unstructured references?

The general trends are the same. For both structured and unstructured references, SBMV outperforms the legacy approach in F1, achieving worse precision and better recall. This tells us that our legacy algorithms are more strict and as a result they miss some links.

Structured reference matching seems easier than unstructured reference matching. The reason is that when we have the structure, we can compare the input reference to the candidate field by field, which is more precise than using the unstructured string.

Structured matching, however, in practise brings new challenges. One big problem is data sparsity. 15% of structured references without DOIs have fewer than four metadata fields. This is not always enough to identify the DOI. Also, 76% of the structured references without DOIs do not contain the article title, which poses a problem for candidate selection using the search engine.

What’s next?

So far, I have focused on evaluating SBMV for unstructured and structured references separately. 17% of the open/limited references at Crossref, however, have both unstructured and structured form. In those cases, it might be beneficial to use the information from both forms. I plan to perform some experiments on this soon.

The data and code for this evaluation can be found at https://github.com/CrossRef/reference-matching-evaluation. The Java version of SBMV (for both structured and unstructured references) can be found at https://gitlab.com/crossref/search-based-reference-matcher.

Reference matching: for real this time

Dominika Tkaczyk — Tue, 18 Dec 2018 00:00:00 +0000

In my previous blog post, Matchmaker, matchmaker, make me a match, I compared four approaches for reference matching. The comparison was done using a dataset composed of automatically-generated reference strings. Now it’s time for the matching algorithms to face the real enemy: the unstructured reference strings deposited with Crossref by some members. Are the matching algorithms ready for this challenge? Which algorithm will prove worthy of becoming the guardian of the mighty citation network? Buckle up and enjoy our second matching battle!

TL;DR

I evaluated and compared four reference matching approaches: the legacy approach based on reference parsing, and three variants of search-based matching.
The dataset comprises 2,000 unstructured reference strings from the Crossref metadata.
The metrics are precision and recall calculated over the citation links. I also use F1 as a standard single-number metric that combines precision and recall, weighing them equally.
The best variant of search-based matching outperforms the legacy approach in F1 (96.3% vs. 92.5%), with the precision worse by only 0.9% (98.09% vs. 98.95%), and the recall better by 8.9% (94.56% vs. 86.85%).
Common causes of SBMV’s errors are: incomplete/erroneous metadata of the target documents, and noise in the reference strings.
The results reported here generalize to the subset of references in Crossref that are deposited without the target DOI and are present in the form of unstructured strings.

Introduction

In reference matching, we try to find the DOI of the document referenced by a given input reference. The input reference can have a structured form (a collection of metadata fields) and/or an unstructured form (a string formatted in a certain citation style).

In my previous blog post, I used reference strings generated automatically to compare four matching algorithms: Crossref’s legacy approach based on reference parsing and three variations of search-based matching. The best algorithm turned out to be Search-Based Matching with Validation (SBMV). SBMV uses our REST API’s bibliographic search function to select the candidate target documents, and a separate validation-scoring procedure to choose the final target document. The legacy approach and SBMV achieved very similar average precision, and SBMV was much better in average recall.

This comparison had important limitations, which affect the interpretation of these results.

First of all, the reference strings in the dataset might be too perfect. Since they were generated automatically from the Crossref metadata records, any piece of information present in the string, such as the title or the name of the author, will exactly match the information in Crossref’s metadata. In such a case, a matcher comparing the string against the record can simply apply exact matching and everything should be fine.

In real life, however, we should expect all sorts of errors and noise in the reference strings. For example, a string might have been manually typed by a human, so it can have typos. The string might have been scraped from the PDF file, in which case it could have unusual unicode characters, ligatures or missing and extra spaces. A string can also have typical OCR errors, if it was extracted from a scan.

These problems are typical for messy real-life data, and our matching algorithms should be robust enough to handle them. However, when we evaluate and compare approaches using the perfect reference strings, the results won’t tell us how well the algorithms handle harder, noisy cases. After all, even if you repeatedly win chess games against your father, it doesn’t mean you will likely defeat Garry Kasparov (unless, of course, you are Garry Kasparov’s child, in which case, please pass on our regards to your dad!).

Even though I attempted to make the data more similar to the noisy real-life data by simulating some of the possible errors (typos, missing/extra spaces) in two styles, this might not be enough. We simply don’t know the typical distribution of the errors, or even what all the possible errors are, so our data was probably still far from the real, noisy reference strings.

The differences in the distributions are a second major issue with the previous experiment. To build the dataset, I used a random sample from Crossref metadata, so the distribution of the cited item types (journal paper, conference proceeding, book chapter, etc.) reflects the overall distribution in our collection. However, the distribution in real life might be different if, for example, journal papers are on average cited more often than conference proceedings.

Similarly, the distribution of the citation styles is most likely different. To generate the reference strings, I used 11 styles distributed uniformly, while the real distribution most likely contains more styles and is skewed.

All these issues can be summarized as: the data used in my previous experiment is different from the data our matching algorithms have to deal with in the production system. Why is this important? Because in such a case, the evaluation results do not reflect the real performance in our system, just like the child’s score on the math exam says nothing about their score on the history test. We can hope my previous results accurately showed the strengths and weaknesses of each algorithm, but the estimations could be far off.

So, can we do better? Sure!

This time, instead of automatically-generated reference strings, I will use real reference strings found in the Crossref metadata. This will give us a much better picture of the matching algorithms and their real-life performance.

Evaluation

This time the evaluation dataset is composed of 2,000 unstructured reference strings from the Crossref metadata, along with the target true DOIs. The dataset was prepared mostly manually:

First, I drew a random sample of 100,000 metadata records from the system.
Second, I iterated over all sampled items, and extracted those unstructured reference strings, that do not have the DOI provided by the member.
Next, I randomly sampled 2,000 reference strings.
Finally, I assigned a target DOI (or null) to each reference string. This was done by verifying DOIs returned by the algorithms and/or manual searching.

The metrics this time are based on the citation links. A citation link points from the reference (or the document containing the reference) to the referenced (target) document.

When we apply a matching algorithm to a set of reference strings in our collection, we get a set of citation links between our documents. I will call those citation links returned links.

On the other hand, in our collection we have real, true links between the documents. In the best-case scenario, the set of true links and the set of returned links are identical. But we don’t live in a perfect world and our matching algorithms make mistakes.

To measure how close the returned links are to the true links, I used precision, recall and F1. This time they are calculated over all citation links in the dataset. More specifically:

Precision is the fraction of the returned links that are correct. Precision answers the question: if I see a citation link A->B in the output of a matcher, how certain can I be that paper A actually cites paper B?
Recall is the percentage of true links that were returned by the algorithm. Recall answers the question: if paper A cites paper B and B is in the collection, how certain can I be that the matcher’s output contains the citation link A->B?
F1 is the harmonic mean of precision and recall.

In the previous experiment, I also used precision, recall and F1, but they were calculated for each target document and then averaged. This time precision, recall and F1 are not averaged but simply calculated over all citation links. This is a more natural approach, since now the dataset comprises isolated reference strings rather than target documents, and in practice each target document has at most one incoming reference.

I tested the same four approaches as before:

the legacy approach, based on reference parsing
SBM with a simple threshold, which searches for the reference string in the search engine and returns the first hit, if its relevance score exceeds the predefined threshold
SBM with a normalized threshold, which searches for the reference string in the search engine and returns the first hit, if its relevance score divided by the string length exceeds the predefined threshold
SBMV, which first applies SBM with a normalized threshold to select a number of candidate items, and a separate validation procedure is used to select the final target item

All the thresholds are parameters which have to be set prior to the matching. The thresholds used in the experiments were chosen using a separate dataset, as the values maximizing the F1 of each algorithm.

Results

The plot shows the overall results of all tested approaches:

The exact values are also given in the table (the best result for each metric is bolded):

	precision	recall	F1
legacy approach	0.9895	0.8685	0.9251
SBM (simple threshold)	0.8686	0.8191	0.8431
SBM (normalized threshold)	0.7712	0.9121	0.8358
SBMV	0.9809	0.9456	0.9629

As we can see, the legacy approach is the best in precision, slightly outperforming SBMV. In recall, SBMV is clearly the best, which also decided about its victory over the legacy approach in F1.

How do these results compare to the results from my previous blog post? The overall trends (the legacy approach slightly outperforms SBMV in precision, and SBMV outperforms the legacy approach in recall and F1) are the same. The most important differences are: 1) on the real dataset SBM without validation is worse than the legacy approach, and 2) this time the algorithms achieved much higher recall. These differences are most likely related to the difference in data distributions explained before.

SBMV’s strengths and weaknesses

Let’s look at a few example cases where SBMV successfully returned the correct DOI, while the legacy approach failed.

Lundqvist D, Flykt A, Ohman A: The Karolinska Directed Emotional Faces - KDEF, CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet. 1998

matched to https://doi-org.turing.library.northwestern.edu/10.1037/t27732-000

The target item is a dataset, which means unusual metadata fields and an unusual reference string.

Schminck, A. , ‘The Beginnings and Origins of the “Macedonian” Dynasty’ in J. Burke and R. Scott , eds., Byzantine Macedonia: Identity, Image and History (Melbourne, 2000), 61–8.

matched to https://doi-org.turing.library.northwestern.edu/10.1163/9789004344730_006

This is an example of a book chapter. The reference string contains special quotes and dash characters.

R. Schneider,On the Aleksandrov-Fenchel inequality, inDiscrete Geometry and Convexity (J. E. Goodman, E. Lutwak, J. Malkevitch and R. Pollack, eds.), Annals of the New York Academy of Sciences440 (1985), 132–141.

matched to https://doi-org.turing.library.northwestern.edu/10.1111/j.1749-6632.1985.tb14547.x

In this case, spaces are missing in the reference string, which might be problematic for the parsing.

R. B. Husar andE. M. Sparrow, Int. J. Heat Mass Transfer11, 1206 (1968).

matched to https://doi-org.turing.library.northwestern.edu/10.1016/0017-9310(68)90036-7

This is another example of a reference string with missing spaces.

F. Cappello, A. Geist, W. Gropp, S. Kale, B. Kramer, and M. Snir. Toward exascale resilience: 2014 update. Supercomputing frontiers and innovations, 1(1), 2014.

matched to https://doi-org.turing.library.northwestern.edu/10.14529/jsfi140101

In this case authors are missing in the Crossref metadata.

Li KZ, Shen XT, Li HJ, Zhang SY, Feng T, Zhang LL. Ablation of the Carbon/carbon Composite Nozzle-throats in a Small Solid Rocket Motor[J]. Carbon, 2011, 49: 1 208–1 215

matched to https://doi-org.turing.library.northwestern.edu/10.1016/j.carbon.2010.11.037

Here we have unexpected spaces inside page numbers.

N. Kaloper, A. Lawrence and L. Sorbo, An Ignoble Approach to Large Field Inflation, JCAP 03 (2011) 023 [ arXiv:1101.0026 ] [ INSPIRE ].

matched to https://doi-org.turing.library.northwestern.edu/10.1088/1475-7516/2011/03/023

In this case we have an acronym of the journal name and additional arXiv id.

KrönerE. ?Stress space and strain space continuum mechanics?, Phys. Stat. Sol. (b), 144 (1987) 39?44.

matched to https://doi-org.turing.library.northwestern.edu/10.1002/pssb.2221440104

This reference string has a missing space, a missing word in the title, and incorrectly encoded special characters.

Suyemoto K. L., (1998) The functions of self-mutilationClinical Psychology Review 18(5): 531–554

matched to https://doi-org.turing.library.northwestern.edu/10.1016/s0272-7358(97)00105-0

In this case the space is missing between the title and the journal name.

Ono , N. 2011 Stable and fast update rules for independent vector analysis based on auxiliary function technique Proceedings of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics 189 192

matched to https://doi-org.turing.library.northwestern.edu/10.1109/aspaa.2011.6082320

The parsing can also have problems with missing punctuation, like in this case.

Hybertsen M.S., Witzigmann B., Alam M.A., Smith R.K. (2002) 1 113

matched to https://doi-org.turing.library.northwestern.edu/10.1023/a:1020732215449

In this case both title and journal name are missing from the reference string.

We can see from these examples that SBMV is fairly robust and able to deal with a small amount of noise in the metadata and reference strings.

What about the errors SBMV made? From the perspective of citation links, we have two types of errors:

False positives: incorrect links returned by the algorithm.
False negatives: links that should have been returned but weren’t.

When we apply SBMV instead of the legacy approach, the fraction of false positives within the returned links increases from 1.05% to 1.91%, and the fraction of false negatives within the true links decreases from 13.15% to 5.44%. This means with SBMV:

1.91% of the links in the algorithm’s output are incorrect
5.44% of the true links are not returned by the algorithm

We can also classify all the references in the dataset into several categories, based on the values of true and returned DOIs:

We have the following categories:

References matched to correct DOIs (1129 cases, returned and true blue)
References correctly not matched to anything (791 cases, returned and true white)
References not matched to anything, when they should be (58 cases, returned white, true grey)
References matched to wrong DOIs (7 cases, returned red, true yellow)
References matched to something, when they shouldn’t be matched to anything (15 cases, returned black, true white)

Note that in terms of these categories, precision is equal to:

And recall is equal to:

What are the most common causes of SBMV’s errors?

Incomplete or incorrect Crossref metadata. Even a perfect reference string formatted in the most popular citation style will not be matched, if the target record in the Crossref collection has many missing or incorrect fields.
Similarly, missing or incorrect information in the reference string is very problematic for the matchers.
Errors/noise in the reference string, such as:
- HTML/XML markup not stripped from the string
- multiple references mixed in one string
- spacing issues and typos
In a few cases a document related to the real target was matched, such as the book instead of its chapter, or the conference proceedings paper instead of the thesis.

Limitations

The most important limitation is the size of the dataset. Every item had to be verified manually, which significantly limited the possibility of creating a large set and also using a lot of independent sets.

Finally, the numbers reported here still don’t reflect the overall precision and recall of the current links in the Crossref metadata. This is because:

we still use the legacy approach for matching,
some references are deposited along with the target DOIs and are not matched by Crossref, these links are not analyzed here, and
in Crossref we have both unstructured and structured references, and in this experiment only the unstructured ones were tested.

What’s next?

The next experiment will be related to the structured references. Similarly as here, I will try to estimate the performance of the search-based matching approach and compare it to the performance of the legacy approach.

The evaluation framework, evaluation data and experiments related to the reference matching are available in the repository https://github.com/CrossRef/reference-matching-evaluation. Future experiments will be added there as well.

https://github.com/CrossRef/reference-matching-evaluation also contains the Python implementation of the SBMV algorithm. The Java implementation of SBMV is available in the repository https://gitlab.com/crossref/search_based_reference_matcher.

Matchmaker, matchmaker, make me a match

Dominika Tkaczyk — Mon, 12 Nov 2018 00:00:00 +0000

Matching (or resolving) bibliographic references to target records in the collection is a crucial algorithm in the Crossref ecosystem. Automatic reference matching lets us discover citation relations in large document collections, calculate citation counts, H-indexes, impact factors, etc. At Crossref, we currently use a matching approach based on reference string parsing. Some time ago we realized there is a much simpler approach. And now it is finally battle time: which of the two approaches is better?

TL;DR

I evaluated and compared four approaches to reference matching: the legacy approach based on reference parsing, and three variants of the new idea called search-based matching.
A large automatically generated dataset was used for the experiments. It is composed of 7,374 metadata records from the Crossref collection, each of which was formatted automatically into reference strings using 11 citation styles.
The main metrics used for the evaluation are precision and recall. I also use F1 as a standard metric that combines precision and recall into a single number, weighing them equally. All values are calculated for each metadata record separately and averaged over the dataset.
In general, search-based matching is better than the legacy approach in F1 and recall, but worse in precision.
The best variant of search-based matching outperforms the legacy approach in average F1 (84.5% vs. 52.9%), with the average precision worse by only 0.1% (99.2% vs 99.3%), and the average recall better by 88% (79.0% vs. 42.0%).
The best variant of search-based matching also outperforms the legacy approach in average F1 for each one of the 11 styles.
A weak spot of the parsing-based approach is degraded/noisy reference strings, which do not appear to use any of the known citation styles.
A weak spot of search-based approach is short reference strings, and in particular citation styles that do not include the title in the reference string.

Introduction

In reference matching, on the input we have a bibliographic reference. It can have the form of an unstructured string, such as:

(1) Adamo, S. H.; Cain, M. S.; Mitroff, S. R. Psychological Science 2013, 24, 2569–2574.

The input can also have the form of a structured reference, such as (BibTex format):

 @article{adamo2013,
 author = {Stephen H. Adamo and Matthew S. Cain and Stephen R. Mitroff},
 title = {Self-Induced Attentional Blink: A Cause of Errors in Multiple-Target Search},
 journal = {Psychological Science},
 volume = {24},
 number = {12},
 pages = {2569-2574},
 year = {2013}
}

The goal of matching is to find the document, which the input reference points to.

Matching algorithms

Matching references is not a trivial task even for a human, not to mention the machines, which are still a bit less intelligent than us (or so they want us to believe…). A typical meta-approach to reference matching might be to score the similarity between the input reference and the candidate target documents. The document most similar to the input is then returned as the target.

Of course, still a lot can go wrong here. We can have more than one potential target record with the same score (which one do we choose?). We can have only documents with low to medium scores (is the actual target even present in our collection?). We can also have errors in the input string (are the similarity scores robust enough?). Life’s tough!

The main difference between various matching algorithms is in fact how the similarity is calculated. For example, one idea might be to compare the records field by field (how similar is the title/author/journal in the input reference to the title/author/journal of our candidate target record?). This is roughly how the matching works currently at Crossref.

The main problem with this approach is that it requires a structured reference, and in practise, often all we have is a plain reference string. In such a case we need to extract the metadata fields from the reference string (this is called parsing). Parsing introduces errors, since no parser is omniscient. The errors propagate further and affect the scoring… you get the picture.

Luckily, as we have known for some time now, this is not the only approach. Instead of comparing structured objects, we could calculate the similarity between them using their unstructured textual form. This effectively eliminates the need for parsing, since the unstructured form is either already available on the input or can be easily generated from the structured form.

What about the similarity scores? We already know a powerful method for scoring the similarities between texts. Those are (you guessed it!) scoring algorithms used by search engines. Most of them, including Crossref’s, do not need a structured representation of the object, they are perfectly happy with just a plain text query.

So all we need to do is to pass the original reference string (or some concatenation of the reference fields, if only a structured reference is available) to the search engine and let it score the similarity for us. It will also conveniently sort the results so that it is easy to find the top hit.

Evaluation

So far so good. But which strategy is better? Is it better to develop an accurate parser, or just rely on the search engine? I don’t feel like guessing. Let’s try to answer this using (data) science. But first, we need to decompose our question into smaller pieces.

Question 1. How can I measure the quality of a reference matcher?

Generally speaking, this can be done by checking the resulting citation links. Simply put, the better the links, the better the matching approach must have been.

A few standard metrics can be applied here, including accuracy, precision, recall and F1. We decided to calculate precision, recall and F1 separately for each document in the dataset, and then average those numbers over the entire dataset.

When I say “documents”, I really mean “target documents”:

precision for a document X tells us, what percentage of links to X in the system are correct
recall for a document X tells us, what percentage of true links to X are present in the system
F1 is the harmonic mean of precision and recall

F1 is a single-number metric combining precision and recall. In F1 precision and recall are weighted equally. It is also possible to combine precision and recall using different weights, to place more emphasis on one of those metrics.

We decided to look at links from the target document’s perspective, because this is what the academic world cares about (i.e. how accurate the citation counts of academic papers are).

Calculating separate numbers for individual documents and averaging them within a dataset is the best way to have reliable confidence intervals (which makes the whole analysis look much smarter!).

Question 2. Which approaches should be compared?

In total we tested four reference matching approaches.

The first approach, called the legacy approach, is the approach currently used in Crossref ecosystem. It uses a parser and matches the extracted metadata fields against the records in the collection.

The second approach is the search-based matching (SBM) with a simple threshold. It queries the search engine using the reference string and returns the top hit from the results, if its relevance score exceeds the threshold.

The third approach is the search-based matching (SBM) with a normalized threshold. Similarly as in the simplest SBM, in this approach we query the search engine using the reference string. In this case the first hit is returned if its normalized score (the score divided by the reference length) exceeds the threshold.

Finally, the fourth approach is a variation of the search based matching, called search-based matching with validation (SBMV). In this algorithm we use additional validation procedure on top of SBM. First, SBM with a normalized threshold is applied and the search results with the scores exceeding the normalized threshold are selected as candidate target documents. Second, we calculate validation similarity between the input string and each of the candidates. This validation similarity is based on the presence of the candidate record’s metadata fields (year, volume, issue, pages, the last name of the first author, etc.) in the input reference string, as well as the relevance score returned by the search engine. Finally, the most similar candidate is returned as the final target document, if its validation similarity exceeds the validation threshold.

By adding the validation stage to the search-based matching we make sure that the same bibliographic numbers (year, volume, etc.) are present in both the input reference and the returned document. We also don’t simply take the first result, but rather use this validation similarity to choose from results scored similarly by the search engine.

All the thresholds are parameters which have to be set prior to the matching. The thresholds used in these experiments were chosen using a separate dataset, as the values maximizing the F1 of each algorithm.

Question 3. How to create the dataset?

Results

We could try to calculate our metrics for every single document in the system. Since we currently have over 100M of them, this would take a while, and we already felt impatient…

A faster strategy was to use sampling with all the tools statistics was so generous to provide. And this is exactly what we did. We used a random sample of 2500 items from our system, which is big enough to give reliable results and, as we will see later, produces quite narrow confidence intervals.

Apart from the sample, we needed some input reference strings. We generated those automatically by formatting the metadata of the chosen items using various citation styles. (Similarly to what happens when you automatically format the bibliography section for your article. Or at least we hope you don’t produce those reference strings manually…)

For each record in our sample, we generated 11 citation strings, using the following styles:

Well known citation styles from various disciplines:
- american-chemical-society (acs)
- american-institute-of-physics (aip)
- elsevier-without-titles (ewt)
- apa
- chicago-author-date
- modern-language-association (mla)
Known styles + random noise. To simulate not-so-clean data, we randomly added noise (additional spaces, deleted spaces, typos) to the generated strings of the following styles:
- american-institute-of-physics
- apa
Custom degraded “styles”:
- degraded: a simple concatenation of authors’ names, title, container title, year, volume, issue and pages
- one author: a simple concatenation of the first author’s name, title, container title, year, volume, issue and pages
- title scrambled: same as degraded, but with title words randomly shuffled

Some styles include the DOI in the reference string. In such cases we stripped the DOI from the string, to make the matching problem non-trivial.

An ideal matching algorithm will match every generated string to the record it was generated from. In practise, some of the expected matches will be missing, which will lower the recall of the tested matching approach. On the other hand, it is very probable that we will get the precision of 100%. To have the precision lower than 100%, we would have to have some unexpected matches to our sampled documents, which is unlikely. This is obviously not great, because we are missing a very important piece of information.

What can we do to “encourage” such mismatches to our sampled documents? We could generate additional reference strings of documents that are not in our sample, but are similar to the documents in our sample. Hopefully, we will see some incorrect links from those similar strings to our sampled documents.

For each sampled document I added up to 2 similar documents (I used, surprise surprise, our search engine to find the most similar documents). I ended up with 7,374 items in total (2,500 originally sampled and 4,874 similar items). For each item, 11 different reference strings were generated. Each reference string was then matched using the tested approaches and I could finally look at some results.

Results

First, let’s compare the overall results averaged over the entire dataset:

The small vertical black lines at the top of the boxes show the confidence intervals at the confidence level 95%. The table gives the exact values and the same confidence intervals. The best result for each metric is bolded.

	average precision	average recall	average F1
legacy approach	0.9933 (0.9910 - 0.9956)	0.4203 (0.4095 - 0.4312)	0.5289 (0.5164 - 0.5413)
SBM (simple threshold)	0.9890 (0.9863 - 0.9917)	0.7127 (0.7021 - 0.7233)	0.7866 (0.7763 - 0.7968)
SBM (normalized threshold)	0.9872 (0.9844 - 0.9901)	0.7905 (0.7796 - 0.8015)	0.8354 (0.8249 - 0.8458)
SBMV	0.9923 (0.9902 - 0.9945)	0.7902 (0.7802 - 0.8002)	0.8448 (0.8352 - 0.8544)

The confidence intervals given in the table are the ranges, in which it is 95% likely to have the real average precision, recall and F1. For example, we are 95% sure that the real F1 for SBMV in our entire collection is within the range 0.8352 - 0.8544.

As we can see, each metric has a different winner.

The legacy approach is the best in precision. This suggests the legacy approach is quite conservative and outputs a match only if it is very sure about it. This might also result in missing a number of true matches (false negatives).

According to the paired Student’s t-test, the difference between the average precision of the legacy approach and the average precision of the second best SBMV is not statistically significant. This means we cannot rule out that this difference is simply the effect of the randomness in sampling, and not the sign of the true difference.

SBM with a normalized threshold is the best in recall. This suggests that it is fairly tolerant and returns a lot of matches, which might also result in returning more incorrect matches (false positives). Also in this case the difference between the winner and the second best (SBMV) is not statistically significant.

SBMV is the best in F1. This shows that this approach balances precision and recall the best, despite being only the second best in both of those metrics. According to the paired Student’s t-test, the difference between SBMV and the second best approach (SBM with a normalized threshold) is statistically significant.

All variants of the search-based matching outperform the parsing-based approach in terms of F1, with statistically significant differences. This shows that in search based-matching it is possible to keep precision almost as good as in the legacy approach, and still include many more true positives.

Let’s also look at the same results split by the citation style:

For all styles the precision values are very high, and the legacy approach is slightly better than all variations of the search-based approach.

In terms of recall and F1 SBM with a simple threshold is better than the legacy approach in 8 out of 11 styles. The three styles for which the legacy approach outperforms SBM with a simple threshold are styles that do not include the title in the reference strings (acs, aip and ewt). The reason for this is that the simple threshold cannot be well calibrated for shorter and longer reference strings at the same time.

SBM with a normalized threshold and SBMV is better than the legacy approach in recall and F1 for all 11 styles.

The weak spot of the legacy approach is degraded and noisy reference strings, which do not appear to use any of the known citation styles.

The weak spot of the search-based matching is short reference strings, and in particular citation styles that do not include the title in the string.

Limitations

The limitations are related mostly to the method of building the dataset.

All the numbers reported here are estimates, since they were calculated on a sample.
The numbers show strengths and weaknesses of each approach, but they do not reflect the real precision and recall in the system:
- Since we included only 2 similar documents for each document in the sample, precision is most likely lower in the real data.
- We used a number of styles distributed uniformly. Of course in the real system the styles and their distribution might be different, which affects all the calculated numbers.