R&D on Crossref

Testing times

Martin Eve — Wed, 03 Apr 2024 00:00:00 +0000

One of the challenges that we face in Labs and Research at Crossref is that, as we prototype various tools, we need the community to be able to test them. Often, this involves asking for deposit to a different endpoint or changing the way that a platform works to incorporate a prototype.

The problem is that our community is hugely varied in its technical capacity and level of ability when it comes to modifying their platform. Some mega-publishers, for instance, outsource their platforms and so are dependent on third party developers/organisations when they want to make a change. Many smaller publishers, by contrast, use systems such as OJS, which come with Crossref plugins that make life very easy… but that require hard code changes to accommodate prototypes. Such changes are way beyond the technical capacity of most journal editors.

So how can we prototype new ideas and test them? One way is by creating new interstitial interfaces that allow people to manually supplement metadata or register for prototype services. Of course, this requires additional work on behalf of the user. Every time they wish to participate they have to visit an extra web page and re-input details that, surely, were included in the original deposit.

Another way would be for plugin developers to have an advanced option field that allowed end-users to change their deposit endpoint. It would be excellent to see this feature in OJS, Janeway, and also proprietary systems. This would allow us to work with the community to test new prototype mechanisms, without forcing anyone to edit code. Many systems already include the ability to switch between Crossref’s “test” system and our live deposit API. All I am really suggesting here is the logical next step: allow advanced users to specify a deposit endpoint of their own choosing so that we can give them access to prototype systems.

Of course, it’s not always that simple. Sometimes, prototype systems will require new data fields on submission, for example. In those cases, there is nothing for it except to modify the plugin or to provide a separate interface. But sometimes, as in the case of the Op Cit project (more on which soon), all the data is already in place; we just need to direct users to a different endpoint. Such changes would definitely make testing times less trying.

Credential Checking at Crossref

Martin Eve — Fri, 15 Mar 2024 00:00:00 +0000

It turns out that one of the things that is really difficult at Crossref is checking whether a set of Crossref credentials has permission to act on a specific DOI prefix. This is the result of many legacy systems storing various mappings in various different software components, from our Content System through to our CRM.

To this end, I wrote a basic application, credcheck, that will allow you to test a Crossref credential against an API.

There are two modes of usage. First, a command-line interface that allows you to run a basic command and get feedback:

Usage: cli.py [OPTIONS] USERNAME PASSWORD DOI

Second, you can use it as a programmatic library in Python:

import cred
credential = cred.Credential(username=username, password=password, doi=doi)

if not credential.is_authenticated():
…

if credential.is_authorised():
…

The tool splits down authentication (whether the given username and password are valid) and authorisation (whether the valid credentials are usable against a specific DOI/prefix).

For technical information, the way this works is by attempting to run a report on the specific DOI in question and then scraping the response page. We hope, at some future point, that there will be a real API for this, but for now this solves the problem as a bridge.

Feedback on automatic digital preservation and self-healing DOIs

Martin Eve — Thu, 28 Sep 2023 00:00:00 +0000

Thank you to everyone who responded with feedback on the Op Cit proposal. This post clarifies, defends, and amends the original proposal in light of the responses that have been sent. We have endeavoured to respond to every point that was raised, either here or in the document comments themselves.

We strongly prefer for this to be developed in collaboration with CLOCKSS, LOCKSS, and/or Portico, i.e. through established preservation services that already have existing arrangements in place, are properly funded, and understand the problem space. There is low level of trust in the Internet Archive, also given a number of ongoing court cases and erratic behavior in the past. People are questioning the sustainability and stability of IA, and given it is not funded by publishers or other major STM stakeholders there is low confidence in IA setting their priorities in a way that is aligned with that of the publishing industry.

We acknowledge that some of our members have a low level of trust in The Internet Archive, but many of our (primarily open access members) work very closely with the IA and our research has shown that, without the IA, the majority our smaller open access members would have almost no preservation at all. We have already had conversations with CLOCKSS and Portico about involvement in the pilot and thinking through what a scale-to-production would look like. That said, for a proof-of-concept, the Internet Archive presents a very easy way to get off the ground, with a stable system that has been running for almost 30 years.

This seems to be a service for OA content only, but people wonder for how long. Someone already spotted an internal CrossRef comment on the working doc that suggested “why not just make it default for everything & everyone”, and that raises concern.

The primary audience for this service is small OA publishers that are, at present, poorly preserved. These publishers present a problem for the whole scholarly environment because linking to their works can prove non-persistent if preservation is not well handled. Enhancing preservation for this sector therefore benefits the entire publishing industry by creating a persistent linking environment. We have no plans to make this the “default for everything and everyone” because the licensing challenges alone are massive, but also because it isn’t necessary. Large publishers like Elsevier are doing a good job of digitally preserving their content. We want this service to target the areas that are currently weaker.

Crossref will always respect the content rights of our members. We never force our members to release their content through Crossref that they don’t ask us to release.

The purpose of the Op Cit project is to make it easier for our members to fulfil commitments they already made when they joined Crossref.

Crossref is fundamentally an infrastructure for preserving citations and links in the scholarly record. We cannot do that if the content being cited or linked to disappears.

When signing the Crossref membership agreement, members agree to employ their best efforts to preserve their content with archiving services so that Crossref can continue to link citations to it even in extremis. For example- if they have ceased operations.

Some of our members already do this well. They have already made arrangements with the major archiving providers. They do not need the Op Cit service to help them with archiving. However, the Op Cit service will still help them ensure that the DOIs that they cite continue work. So it will still benefit them even if they don’t use it directly.

However, our research shows that many of our members are not fulfilling the commitments they made when joining Crossref. Over the next few years, we will be trying to fix this. Primarily through outreach- encouraging members to set up and record with Crossref archiving arrangements with the archives of their choice.

But we know some members will find this too technically challenging and/or costly. [And frankly, given what we’ve learned of the archiving landscape, we can see their point.] The proposed Op Cit service is for these members. The vast majority of these members are Open Access publishers, so the “rights” questions are far more straightforward- making the implementation of such a service much more tractable.

Someone asked what this means for the publisher-specific DOI prefix for this content? Will this be lost?

No.

There is concern about the interstitial page that Crossref would build that gives the user access options. The value of Crossref to publishers is adding services that are invisible and beneficial to users, not adding a visible step that requires user action.

There is nothing in Crossref’s terms that says that we have to be invisible. The basic truth is that detecting content drift is really hard and several efforts to do so before have failed. Without a reliable way of knowing whether we should display the interstitial page, which may become possible in future, we have to display something for now, or the preservation function will not work.

Crossref has, also, supported user-facing interstitial services for over a decade, including:

Multiple Resolution
Coaccess
CrossMark
Crossref Metdata Search
REST API

So we have a long track record of non-B2B service provision.

There is confusion about why Crossref seems to want to build the capacity to “lock” records in absence of flexibility. People feel no need for Crossref to get involved here.

This is a misunderstanding of the terminology. The Internet Archive allows the domain owner to request content to be removed. This would mean that, in future, if a new domain owner wanted, they could remove previously preserved material from the archive, thereby breaking the preservation function. When we say we want to “lock” a record, we mean that a future domain owner cannot remove content from the preservation archive. This also prevents domain hijackers from compromising the digital preservation.

There is concern about the possibility to hack this system to give uncontrolled access to all full-text content by attacking publishing systems and making them unavailable. This is an unhappy path scenario but something on people’s minds.

The system only works on content that is provided with an explicitly stated open license (see response above).

I think this project would be improved by better addressing the people doing the preservation maintenance work that this requires. Digital preservation is primarily a labor problem, as the technical challenges are usually easier than the challenge of consistently paying people to keep everything maintained over time. Through that lens, this is primarily a technical solution to offload labor resources from small repositories to (for now) the Internet Archive, where you can get benefits from the economies of scale. There are definitely cases where that could be useful! But I think making this more explicit will further a shared understanding of advantages and disadvantages and help you all see future roadblocks and opportunities for this approach.

This consultation phase was designed, precisely, to ensure that those working in the space could have their say. While this is a technical project, we recognize that any solution must value and understand labor. That means that any scaling to production must and will also include a funding solution to address the social labor challenge.

Is there any sense in polling either the IA Wayback Machine or the LANL Memento Aggregator first to determine if snapshot(s) already exist?

We could do this, but it would add an additional hop/lookup on deposit. Plus, we want to store the specific version deposited at the specific time it is done, including re-deposits.

I would encourage looking at a distributed file system like IPFS (https://en.wikipedia.org/wiki/InterPlanetary_File_System). This would allow easy duplication, switching and peering of preservation providers. Correctly leveraged with IPNS; resolution, version tracking and version immutability also become benefits. Later after beta the IPNS metadata could be included as DOI metadata.

We had considered IPFS for other projects, but really, for this, we want to go with recognised archives, not end up running our own infrastructure for preservation.

It might be useful to look into the 10320/loc option for the Handle server: the https://www.handle.net/overviews/handle_type_10320_loc.html. I can imagine a use case where a machine agent might want to access an archive directly without needing to go to an interstitial page.

It is good to see reference to the HANDLE system and alternative ways that we might use it. We will consult internally on the technical viability of this.

In general, though, we prefer to use web-native mechanisms when they are available. We already support direct machine access via HTTP redirects and by exposing resource URLs in the metadata that can be retreivd via content negotiation. In this case, we would be looking at supporting the 300 (multiple choice) semantics.

I’m curious to see how this will work for DOI versioning mechanisms like in Zenodo, where you have one DOI to reference all versions as well as version specific DOIs. If your record contains metadata + many files and a new version just versions one of the several files my assumption is that within the proposed system an entire new set (so all files) is archived. In theory this could also be a logical package, where simply the delta is stored, but I guess in a distributed preservation framework like the one proposed here, this would be hard to achieve.

This is a good point and it could lead to many more, frustrating, hops before the user reaches the content. We will conduct further research into this scenario, but we also note that Zenodo’s DOIs do not come from Crossref, but from DataCite.

There’s a decent body of research at this point on automated content drift detection. This recent paper: https://ceur-ws.org/Vol-3246/10_Paper3.pdf likely has links to other relevant articles.

We have no illusions about the difficulty of detecting semantic drift but this is helpful and interesting. We will read this material and related articles to appraise the current state of content drift detection.

Out of curiosity, will we be using one type of archive (i.e., IA or CLOCKSS or LOCKSS or whatever) or will it possibly be a combination of a few archives? Reading the comments, it looks like some of them charge a fee, so I see why we’d use open source solutions first. Also, eventually could it be something that the member chooses? i.e. which archive they might want to use. Again, the latter question isn’t something for the prototype, but I’m curious about this use case. Also, I wonder about the implementation details if it is more than one archive. The question is totally moot of course, if we’re sticking with one archive for now.

The design will allow for deposit in multiple archives – and we will have to design a sustainability model that will cover those archives that need funding. As above, this is an important part of the move to production.

Will be good for future interoperability to make sure at least one of the hashes is a SoftWare Hash IDentifier (see swhid.org). The ID is not really software specific and will interoperate with the Software Heritage Archive and git repositories.

We will certainly ensure best practices for checksums.

Comments on the Interstitial Page

I’d keep the interstitial page without planning its eradication. (See why in the last paragraph) I’d even advocate for it to be a beautiful and useful reminder to users that “This content is preserved”. I’d go further and recommend that publishers deposit alternate urls of other preservation agents like PMC etc, that would also be displayed. This page could even be merged with multi-resolution system.

The why: I’m concerned of hackers and of predatory publishers exploiting the spider heuristics by highjacking small journals and keeping just enough metadata as in them as to fool the resolver and then adding links to whatever products, scams and whatnots…

Technical. Scraping landing pages is hard. We’ve had a lot of projects to do this over the years. You can mitigate the risk by tiering / heuristics. Maybe even feedback loop to publishers to encourage them to put the right metadata on the landing page.

This is the only part of this proposal that I don’t like. People are used to DOIs resolving directly to content, and I don’t think that should be changed unless absolutely necessary. I would prefer that the DOI resolves to the publisher’s copy if it exists, and the IA copy otherwise.

We will continue the discussion about the interstitial page. The basic technical fact, as above, is that detecting content drift is hard and so we may need, at least, to start with the page. However, some commentators presented reasons for keeping it.

We also have already supported interstitial pages for multiple resolution and co-access for over a decade.

It is member’s choice whether they wish to deposit alternative URLs and we already have a mechanism for this.

A Request for Comment - Automatic Digital Preservation and Self-Healing DOIs

Martin Eve — Thu, 29 Jun 2023 00:00:00 +0000

Digital preservation is crucial to the “persistence” of persistent identifiers. Without a reliable archival solution, if a Crossref member ceases operations or there is a technical disaster, the identifier will no longer resolve. This is why the Crossref member terms insist that publishers make best efforts to ensure deposit in a reputable archive service. This means that, if there is a system failure, the DOI will continue to resolve and the content will remain accessible. This is how we protect the integrity of the scholarly record.

I will write another post, soon, on the reality of preservation of items with a Crossref DOI, but recent work in the Labs team has determined that we have a situation of drastic under-preservation of much scholarly material that has been assigned a persistent identifier. In particular, content from our smaller Crossref members, with limited financial resources, is often precariously preserved. Further, DOI URLs are not always updated, even when, for instance, the underlying domain has been registered by a different third party. This results in DOIs pointing to new, hijacked, and elapsed content that does not reflect the metadata that we hold.

We (Geoffrey) have (has) long-harboured ambitions to build a system that would allow for automatic deposit into an archive and then to present access options to the resolving user. This would ensure that all Crossref content had at least one archival solution backing it and greatly contribute to the improved persistent resolvability of our DOIs. We refer to this, internally, as “Project Op Cit”. And we’re now in a position to begin building it.

However, we need to get this right from the design phase out. We need input from librarians working in the digital preservation space. We need input from members on whether they would use such a service. We are not digital preservation experts and we are acutely aware that we need the expertise of those who are, particularly where we’ve had to take some shortcuts. For instance: we are aware that the Internet Archive is perhaps not the first choice of many digital preservation librarians and specialists, who opt for specific scholarly-communications solutions. However, it is easy, open, and free. Hence, we propose for the prototype to use IA, on the assumption that this will be a proof-of-concept only, which we will expand to other archives if there is demand and once it works.

So: please do read the below and add your comments and questions to this thread in the community forum (link below), or send me queries/concerns by email. It would be excellent if we could receive comments by mid-August 2023. If you would rather comment on a Google doc, that’s also possible.

If enough people are interested, we could also host a community call to discuss this design and its prototyping. Do please, when emailing, let me know if this is of interest.

Project Op Cit (Self-Healing DOIs)

Request for Comment

This document sets out the problem statement, a proposed prototype solution, and a transition path to production if successful.

Proposed Prototype Solution

For members who opt-in to the service, We have a special class of DOI (only for open-access content) where, when the DOI is registered:

We immediately make an archive of the item with any archiving services that care to participate in the project (minimally, the Internet Archive, which is the easiest for us to begin with, but a modular/pluggable archival system). The Internet Archive Python Library should let us submit to them. We could pursue other arrangements with CLOCKSS, LOCKSS, and Portico.
We update the XML to reflect the archives to which it has been submitted.
The DOI landing page is redirected to an interstitial page that we control. This page gives the user access options.
We develop processes to determine whether the original URL “works”. The heuristics that define whether a resource has changed substantially or works need long-term consideration and real-world testing. Using the interstitial page approach will allow us to refine this, with a long-term goal of eradicating it.

Figure 1: The Deposit Process

Figure 2: The Resolution Process

Potential Challenges

Content drift. It would be extremely difficult to detect content change vs. (eg) page structure change, except in the case of binary fulltext. However, we can poll for the DOI at an HTML endpoint and detect when binary fulltext items, such as a PDF, change.
Latency on resolver if lookup is real-time. For this reason, we need a periodic crawler so that resolvers do not wait for real-time detection on access.
If using Internet Archive, the domain owner (at the present moment) can request the removal of content. We would need the capacity to “lock” records that are being used as Op Cit redirection archival copies. This requires a further conversation with the Internet Archive.

Prototype Components/Architecture

Registration Proxy and Database (“Fleming”)

The registration proxy implements a pass-through to the deposit API and hosts a relational database of self-healing DOIs (Postgres). It will be hosted at api.labs.crossref.org/deposit/opcit and clients will have to use this endpoint to deposit. Simultaneously, the proxy will:

Determine the license status of the incoming item.
If the license is open and fulltext is provided, deposit a copy in selected digital preservation archives. Store proof of licensing attestation.
In the case of binary files (fulltext PDF), store a hash of the content.
Store the DOI, binary hash, and all URLs in a relational database under “pending” state.
Pass through the request to Crossref’s content registration system.
Monitor the result of this request and remove stored data if registration fails.
Re-registration through Fleming will update existing entries and re-fix their data against content drift at this time.

Spider (“Shelob”)

A series of components that:

Check that “pending” DOIs have been successfully registered. Remove those that have not and move those that have to “active” state.
Dereference “active” DOIs and ensure that we have the most current URL in case updates have gone directly to the live resolver.
Periodically crawl URLs in the self-healing database.
On HTTP 301 code, update database entry to point to new permanent URL.
On HTTP 302 code, follow the temporary redirect expecting the original content.
On HTTP 4xx codes, mark the entry as dead.
On HTTP 200 code of HTML landing page, parse the page for the presence of the DOI. If the DOI is not present, mark the entry as dead.

Resolver Proxy (“Hippocrates”)

Display an interstitial landing page with archival versions and an explanation.
At some future point, for active entries, resolve to the stored URL (faster but could be de-synced) or pass the request to the live resolver (requires an extra hop but will always be in-sync with deposit).

Observability and statistics

Metrics we will collect:

Count of DOIs using Op Cit
Count of visitors arriving on Op Cit landing pages
Usage count of each outgoing link/access option

A daily report will present:

Newly “failed” entries that we believe have died
These will be checked extensively, particularly at first, to ascertain whether our failure heuristics are valid
Entries that have recovered

Errors will be logged and monitored via Grafana.

Documentation and Automated Tests

Core assumptions and new behaviours of the platform will be documented as part of the prototype.
Automated tests will be written, especially for the spider (“Shelob”), which must handle a diverse variety of real-world situations.

Prototype Architecture Requirements

Postgres RDS for resolution/self-healing DOI data (AWS).
FastAPI hosting for passthrough proxy (fly.io).
EC2 hosting for the spider (AWS).
FastAPI hosting for resolver proxy (fly.io).

Transition to Production

If this prototype garners popular appeal, a transition to production would need to keep some prototype components and rewrite others.

“Fleming” would need to be rewritten as a deposit module / integrated with Manifold’s (the next-generation system at Crossref) deposit. If this would create too much overhead, it need not be a blocking process in the deposit.
“Shelob” would continue to need to run continuously and to scale with the adoption of self-healing DOIs unless one of the other options were used.
Prototype architecture will be written so that spidering can be distributed between several servers, if required.
“Hippocrates” would need to be integrated into the live link resolver. Depending on how a field for a self-healing DOI is embedded in Manifold, this may not need any additional database hits.

Back Content

We also have a database of back content stored by the Internet Archive, mapped to DOIs where they have been able to do so. This data source could be used to enable self-healing DOIs on all content in this archive.

Crossref Research and Development: Releasing our Tools from the Ground Up

Martin Eve — Wed, 21 Jun 2023 00:00:00 +0000

This is the first post in a series designed to showcase what we do in the Crossref R&D group, also known as Crossref Labs, which over the last few years has been strengthened, first with Dominika Tkaczyk and Esha Datta, last year with part of Paul Davis’s time, and more recently, yours truly. Research and development are, obviously, crucial for any organisation that doesn’t want to stand still. The R&D group builds prototypes, experimental solutions, and data-mining applications that can help us to understand our member base, in the service of future evolution of the organisation. One of the strategic pillars of Crossref is that we want to contribute to an environment in which the scholarly research community identifies shared problems and co-creates solutions for broad benefit. We do this in all teams through research and engagement with our expanding community.

For example, if the metadata team wants to implement a new field in our schema, it helps to have a prototype to show to members. The Labs team would implement such a prototype. If we want to know the answer to a question about the 150m or so metadata records we have – e.g. how many DOIs are duplicates? – it’s the Labs team that will work on this.

When building such prototypes, which can often seem esoteric and one-off, though, it can be easy to believe that there is no way anybody else would re-use our components. At the same time, we find ourselves consistently working with the same infrastructures, re-using different code blocks across many applications. One of the tasks I have been working on is to extract these duplicated functions and to get them into external code libraries.

Why is this important? As many readers doubtless know, Crossref is committed to The Principles of Open Scholarly Infrastructure. For reasons of insurance, everything we do and newly develop is open source and we want our members to be able to re-use the software that we create. It’s also important because, if we centralize these low-level building blocks, we make it much easier to fix bugs when they occur, which would otherwise be distributed across all of our projects.

As a result, Crossref Labs has a series of small code libraries that we have released for various service interactions. We often find ourselves needing to interact with AWS services. Indeed, Crossref’s live systems are in the process of transitioning to running in the cloud, rather than our own data centre. It makes sense, therefore, for prototype Labs systems to run on this infrastructure, too. However, the boto3 library is not terribly Pythonic. As a result, many of our low-level tools interact with AWS. These include:

CLAWS: the Crossref Labs Amazon Web Services toolkit. The CLAWS library gives speedy and Pythonic access to functions that we use again and again. This includes downloading files from and pushing data to S3 buckets (often in parallel/asynchronously), fetching secrets from AWS Secrets Manager, generating pre-signed URLs, and more.
Longsight: A range of common logging functions for the observability of Python AWS cloud applications. Less mature than CLAWS, this is the starting point for observability across Labs applications. It supports running in AWS Lambda function contexts or pushing your logs to AWS Cloudwatch from anywhere else. It also supports logging metrics in structured forms. Crucially, the logs are all converted into machine-readable JSON format. This allows us to export the metrics into Grafana dashboards to visualize failure and performance.
Distrunner: decentralized data processing on AWS services. Easily the least mature and experimental of these libraries, distrunner is one of the ways that we distribute the workloads of our recurrent data processing. A number of the Labs projects require us to run recurrent data-processing tasks. For instance, my colleague Dominika Tkaczyk has developed the sampling framework that is regenerated once per week. We use Apache Airflow (and, specifically, Amazon Managed Workflows for Apache Airflow) to host these periodic tasks. This is useful because it gives us quick, visual oversight if tasks fail. However, the Airflow worker instances on AWS are quite severely underpowered and unsuitable for large in-memory activities. Hence, the sampling framework fires up a Spark instance for its processing. Often, though, we do not need the parallelization of Spark and just want to be able to run a generic Python script in a more powerful environment. That’s what distrunner is designed to do. The current version uses Coiled but this may change in the future.

While these tools will be useful to nobody except programmers – and this has been quite a technical post – there is a broader philosophical point to be made about this approach, in which everything is available for re-use, “from the ground up”. The point is: we also try, in Labs and in the process of “R&Ding”, to work without privileged access. That is: I don’t get “inside” access to a database that isn’t accessible to external users. I have to work with the same APIs and systems as would an end-user of our services. This means that, when we develop internal libraries, it’s worth releasing them. Because they use systems that are accessible to any of our users.

I should also say that our openness is more than unidirectional. While we are putting a lot of effort into ensuring that everything new we put out is openly accessible, we are also open to contributions coming in. If we’ve built something and you make changes or improve it, please do get in touch or submit a pull request. Openness has to work both ways if projects are truly to be used by the community.

Future posts – coming soon! – will introduce some of the technologies and projects that we have been building atop this infrastructure. This includes a Labs API system; new functionality to retrieve unpaginated datasets of whole API routes; a study of the preservation status of DOI-assigned content; and a mechanism for modeling new metadata fields.

Martin Paul Eve is joining our R&D group as a Principal Developer

Geoffrey Bilder — Fri, 26 Aug 2022 00:00:00 +0000

I’m delighted to say that Martin Paul Eve will be joining Crossref as a Principal R&D Developer starting in January 2023.

As a Professor of Literature, Technology, and Publishing at Birkbeck, University of London- Martin has always worked on issues relating to metadata and scholarly infrastructure. In joining the Crossref R&D group, Martin can focus full-time on helping us design and build a new generation of services and tools to help the research community navigate and make sense of the scholarly record.

Martin himself explains the logic of this move on his own blog, so I won’t attempt to do the same here other than to say:

praxis makes perfect.

(mic drop)

Created with DALL·E, an AI system by OpenAI with the the prompt: ‘A bookwheel in the style of the 16th-century illustration by Agostino Ramelli and where the books are replaced by open laptops’

Announcing our new Head of Strategic Initiatives: Dominika Tkaczyk

Geoffrey Bilder — Fri, 10 Jun 2022 00:00:00 +0000

TL;DR

A year ago, we announced that we were putting the “R” back in R&D. That was when Rachael Lammey joined the R&D team as the Head of Strategic Initiatives.

And now, with Rachael assuming the role of Product Director, I’m delighted to announce that Dominika Tkaczyk has agreed to take over Rachael’s role as the Head of Strategic Initiatives. Of course, you might already know her.

We will also immediately start recruiting for a new Principal R&D Developer to work with Esha and Dominika on the R&D team.

What does this mean for R&D?

Before I talk about what Dominika’s move means in practice, I just want to take a moment to thank Rachael for the time she spent working with us. Over the past year, she has injected a massive amount of energy into the group and rebuilt the team’s momentum. This is exactly what we asked her to do.

Rachael’s first task was to repatriate her two R&D colleagues, who we had loaned to work on other urgent projects. Dominika was the technical lead on the port and relaunching of the REST API. Esha was the technical lead for the ROR initiative. In addition, Rachael has been working with Esha, Dominika, Paul Davis, and me on several shorter-term strategic projects that are shaping our overall development strategy.

Exploring and implementing a new approach to building content registration front ends. This approach is schema-driven and bakes in localization and accessibility support from the start. The new approach is currently the basis for the grant registration tool that our Product & Tech teams are now testing with our new funder members.
Exploring and ultimately rejecting a “pull-based” approach to registering metadata, where Crossref would harvest structured metadata from member landing pages instead of asking members to deposit it with us via XML. You are not really doing R&D unless some of your ideas fail. In this case, we quickly discovered that the logistics of crawling our members’ websites, combined with the sparsity of structured metadata in landing pages, made a pull-based approach fragile and impractical.
Exploring the use of ML techniques to fill gaps in the journal classification data that is currently in the REST API. Gaining new data science badges in the process.
Exploring alternative approaches to building community-extendable reporting tools using standard data science tooling and techniques.
Exploring how we can help reduce support toil by using data science tools like notebooks to create new support tools and self-serve UIs for information frequently requested by members that can otherwise prove difficult to get using our existing tools.
Looking at extending the matching technology previously developed by labs to try and better match funder grant-information research outputs.

And this is just a sample of projects Rachael helped promote and prioritize. It is the nature of many of the larger R&D projects that you don’t see the immediate results until long after they’ve been conceived and put into motion. This means that Rachael has been working on some things over the past year that are not yet public.

But, with any luck, we may see some significant new developments in how Crossref collects and distributes information about significant updates to the scholarly record- including retractions and withdrawals. We are also likely to see more work to promote data citation amongst our members. And finally, we are likely to see an attempt to create a community-managed and open research classification taxonomy. Of course, as is the case with research projects, there is no guarantee that any of these nascent ideas/projects will make it into a production service. Still, if even one of them does, it will become as vital a part of open scholarly infrastructure as DOIs, ORCIDs, or ROR IDs are now.

And we will have Rachael and the hard work of the R&D group, important cameos from others, and community input to thank for giving them the initial push to realization. So that’s a pretty good track record for just a year in the R&D group.

Passing the torch

And this is a track record I’m confident that Dominika can match as she takes over Rachael’s role.

Soon after Dominika joined the Crossref R&D team, she started to expand her activities to include more production engineering practice, team leadership, and community outreach. She has also worked extensively with support and outreach- providing them with data science consulting and mentoring in software development. Her new role as the Head of Strategic Initiatives will continue this trend. She will spend less time prototyping software and analyzing data and more time liaising with our members and the broader community to understand their needs and design R&D projects to test approaches to meeting those needs. This means a lot more liaising with other Crossref teams, speaking with our members and the wider community, and participating in working groups and conferences.

It also probably means a lot less programming and analysis. But programming and building prototypes are critical to the R&D team. And so the first thing we will do is start recruiting for a new Principal R&D Developer to continue working along with Esha on conducting experiments and developing POCs.

I’m looking forward to the next year. With Rachael taking the role of Product Director and Dominika taking over as the Head of Strategic Initiatives, we are well-positioned to make profound technical and conceptual improvements to Crossref’s services while simultaneously working with the community to line up our next strategic priorities.

Follow the money, or how to link grants to research outputs

Dominika Tkaczyk — Tue, 22 Mar 2022 00:00:00 +0000

The ecosystem of scholarly metadata is filled with relationships between items of various types: a person authored a paper, a paper cites a book, a funder funded research. Those relationships are absolutely essential: an item without them is missing the most basic context about its structure, origin, and impact. No wonder that finding and exposing such relationships is considered very important by virtually all parties involved. Probably the most famous instance of this problem is finding citation links between research outputs. Lately, another instance has been drawing more and more attention: linking research outputs with grants used as their funding source. How can this be done and how many such links can we observe?

TL;DR

We looked for links between research outputs and grants registered with Crossref.
Grant DOIs alone are not enough for linking research outputs with grants, because the funding information in research outputs typically does not contain grant DOIs (yet). Award numbers alone are also not enough because they are not globally unique.
We used either grant DOIs (if available) or the combination of award number and funder information to match grants to research outputs.
In total, we found 20,834 links between research outputs and registered grants, involving 17,082 research outputs and 3,858 grants (10% of all registered grants)¹.
Erroneous and incomplete metadata, especially involving award numbers, is the main factor that prevents linking research outputs to grants.

Introduction

The ecosystem of scholarly metadata is filled with relationships between items of various types: a person authored a paper, an author works at a university, a paper cites a book, a book contains a chapter, a funder funded research. Those relationships are absolutely essential: an item without them is missing the most basic context about its structure, origin, and impact.

No wonder that finding and exposing relationships between items in the scientific ecosystem is considered very important by virtually all parties involved. Probably the most famous instance of this problem is finding citation links between research outputs. Another, relatively new example, is linking research outputs with grants used as their funding source.

At Crossref, for some time now we have been seeing a steady growth of funder membership and grant registration. We are aware that the possibility of finding relationships between grants and research outputs is a big reason why funders are registering grants with us in the first place. Being able to see which research outputs are being supported by which grants helps reduce the reporting burden on researchers, funders, and institutions alike, especially now with the addition of ROR IDs to help complete the picture. Exposing relationships between research outputs and grants also increases the transparency of funding sources of the research, making it easier to assess and trust scientific findings.

But how can we find those relationships and how many of them can we already observe? Thankfully our REST API, recently equipped with the grant metadata, can help us answer these questions.

The perfect scenario

Imagine a world where the metadata of any scientific output states all relationships with other items existing in the scientific ecosystem, and those related items are always referred to by their persistent identifiers, allowing all this information to be accessed in a fully machine-readable way… Lovely, right?

In the case of citations, in such a perfect world every bibliographic reference has a DOI of the cited item. And in the case of funding information, a scientific paper contains grant DOIs, stating the funded-by relationships between the paper and the grants.

But, as the last two years have painfully taught us all, life is not all rainbows and unicorns.

The reality kicks in

We know that around 71% of bibliographic references are deposited with Crossref without a DOI of the cited item. This means that if we want to establish citation links between items, we need to match the bibliographic references using the provided metadata, which is not a trivial task and can potentially introduce errors.

And the situation with the funding information and grant DOIs is even worse.

Problem #1: our schema does not allow the publishers to attach grant DOIs to research outputs

This issue is 100% on us. Because grant DOIs are relatively new, our deposit schema does not yet allow to specify the grant DOI in the funding information of a research output, even if the publisher wanted to. We are working on changing this.

Interestingly, it looks like persistent identifiers always find a way. Within over 7.4 million research outputs with funding information, we noticed 6 cases where a grant DOI was provided as an award number. For example in 10.1093/nar/gkaa994 we have the following:

funder: [
{
name: "Wellcome Trust",
award: ["10.35802/108758"],
doi-asserted-by: "publisher",
DOI: "10.13039/100010269"
}, ...
]

This may not be 100% correct from the schema perspective, but it is very useful when one is interested in linking grants to research outputs!

But those cases are extremely rare outliers. For the vast majority of the outputs, grant DOIs are not present in the metadata. This means that, just like in the case of bibliographic references, we have to use the metadata to match funding information to grants.

Funding information is typically given as a pair: award number, funder information. Grants contain similar metadata. One might be tempted to use only the award number for linking, as in some cases it can look like a grant identifier.

Let’s consider an example. We want to find all papers funded by grant 10.37807/gbmf7622. The award number is GBMF7622. A simple approach might be to search for items with this award number in Crossref’s REST API, which returns 12 results². However, one of the resulting items is the grant itself³. So excluding that, it seems like there are 12-1=11 research outputs funded by this grant.

Simple and easy, right? Well, think again.

Problem #2: award numbers are not unique

Let’s look at another example grant: 10.25585/60000600. Its award number is 2817 and the funder is the US Department of Energy.

When we search for this award we get 10 results⁴. Like before, one of them is our grant. After examining the remaining 9 we will see that:

3 items have been funded by the Joint Genome Institute, which according to the Funder Registry has been incorporated into Basic Energy Sciences, which is a descendant of the US Department of Energy
2 items have been funded by International Rett Syndrome Foundation from the US
2 items have been funded by Agencia Nacional de Promoción Científica y Tecnológica from Argentina
1 item has been funded by Arak University of Medical Sciences from Iran
1 item has been funded by Shahrekord University also from Iran

So among only 9 items mentioning the same award number we have in fact 5 different grants. Our input grant should probably be linked only to the three items mentioning Joint Genome Institute. The main problem illustrated here is that the award numbers are not globally unique, and thus should not be treated like identifiers.

Indeed, within 38,326 grants registered so far, we have 37,608 distinct award numbers, and among those, there are 716 award numbers, each of which appears in multiple grants. This issue comes in two flavours: conflicts between and within funders.

Between-funder award number conflicts

A conflict between funders is when more than one funder uses the same award number for one of their grants. This is expected - award numbers are assigned by funders internally and are not designed to be a globally unique identifier.

Out of 716 award numbers that appear in multiple grants, 12 are numbers that appear in grants of different funders. For example, there are two grants with the award number 105626:

Systemic MFG-E8 Blockade as Melanoma Therapy funded by Melanoma Research Alliance
Institutional Strategic Support Fund Phase2 FY2014/16 funded by Wellcome Trust

Because of those conflicts, we cannot simply rely on the award numbers for linking grants to research outputs. Instead, we have to use more information to be sure that the links are correctly established.

Within-funder award number conflicts

To our big surprise, it turns out that the majority of the award number conflicts happen not between different funders, but within the grants of a single funder. Out of 716 award numbers that appear in multiple grants, 704 appear in multiple grants of a single funder only. Such situations are not expected and could indicate an error or some other systematic issue with the data.

Interestingly, out of those 704 award numbers, 700 are associated with the US Department of Energy. We’ve followed up with them in order to clarify or resolve this. The US Department of Energy pointed out a fundamental issue with the data model: currently a grant deposited with Crossref has to have at least one funder DOI, and no other way of identifying the associated organisation is allowed. At the same time, some of the facilities that should appear in their grants’ metadata are not funders at all and thus cannot be identified by a funder DOI. In the future, they plan to identify those facilities in their grant metadata by providing ROR IDs.

Because of within-funder award number conflicts, in some cases it might be difficult to distinguish between two grants with the same award number and funder. A solution might be to use additional information or simply not accept any links if a research output cannot be reliably linked to one grant only.

Our linking approach

Based on all those observations, we adopted the following approach:

We iterated over all registered grants, for each we performed the following steps:
- We used award.number:<grant DOI> filter in the REST API to find all items listing a given grant’s DOI as the award number. Because this is based on the grant’s persistent identifier, we recorded those links without any further verification.
- We used the award.number:<grant award number> filter in the REST API to find all items listing grant’s award number in the funding information. Each resulting item was then verified by comparing the funder information in the item to the funder information in the grant. We recorded the link between the grant and the candidate item only if the verification succeeded.
In the final step, we examined all recorded links to make sure that each pair (research output, award number) is linked to at most one grant. Links violating this rule were flagged as not reliable.

We used different techniques to verify the funder information between the research output (item) and the grant, depending on what information is available. Grants always have the funder DOI. The item, however, can have the funder DOI, the funder name, or both.

If the funder DOI was available on both sides, the following rules were used for the funder verification (ordered by decreasing confidence):

Both the item and the grant contain the same funder DOI, for example, 10.35802/089928 and 10.1242/jcs.196758
The funder in the item replaced or was replaced by the funder in the grant (according to the Funder Registry), for example, 10.35802/104848 and 10.1136/medethics-2020-106821
The funder in the paper is an ancestor or a descendant of the funder in the grant (according to the Funder Registry), for example, 10.46936/sthm.proj.2010.40084/60004575 and 10.1016/j.heliyon.2018.e00629

If the funder DOI was not available in the item, the following rules were used for the funder verification (ordered by decreasing confidence):

The funder name in the paper is the same (ignoring the case) as the funder name in the grant, for example, 10.35802/110166 and 10.12688/wellcomeopenres.14645.4
The funder name in the item is the same (ignoring the case) as the name of the funder that replaced/was replaced by the funder in the grant, for example, 10.35802/206194 and 10.1172/jci.insight.96381
The funder name in the item is the same (ignoring the case) as the name of the ancestor/descendant of the funder in the grant, for example, 10.46936/cpbl.proj.2001.2191/60002922 and 10.1109/tkde.2016.2628180

Note that this is in fact very similar to our reference matching approach. In both cases, first we search for candidate items, and then verify the candidates by comparing the metadata. The actual metadata used for the verification varies, because different information is typically given in the bibliographic reference and the funding information.

What we found

This procedure applied to the entire Crossref dataset resulted in 20,846 links between research outputs and grants⁵. Of those, 12 were flagged as unreliable, because they involved more than one grant linked to the same item and award number. The rest of this section focuses on the remaining 20,834 links.

Within the 20,834 links, we have 17,082 research outputs and 3,858 (10.1%) grants.

Here is the breakdown into the verification approaches used:

Verification	#links	%links
The item contains grant DOI - no verification	6	<0.1%
Funder DOIs are the same	8,364	40.1%
Funder DOIs are related with a replaced/was replaced by relationship	3,704	17.8%
Funder DOIs are related with an ancestor/descendant relationship	7,718	37.0%
Funder names are the same	591	2.8%
The name of the funder in the item is the same as the name of the funder that replaced/was replaced by the funder in the grant	364	1.7%
The name of the funder in the item is the same as the name of the ancestor or descendant of the funder in the grant	87	0.4%

In most cases, just using the funder DOIs for the verification was enough. Verifying by the funder name added 1,042 links, which is 5% of all links.

And here are statistics for individual funders. Only funders with at least 10 deposited grants are listed in the table. The table shows the number of detected links, the number of distinct research outputs linked, the total number of outputs mentioning the given funder DOI, and the number of grants.

Funder	#links	#linked research outputs	#total outputs with funder DOI	#grants
Japan Science and Technology Agency	11,922	10,411	25,779	9,383
Wellcome Trust (including both funder DOIs 10.13039/100004440 and 10.13039/100010269)	8,001	6,246	49,492	17,534
James S. McDonnell Foundation	463	457	2,534	557
Melanoma Research Alliance	152	150	894	392
Asia-Pacific Network for Global Change Research	100	100	838	539
ALS Association	84	78	909	434
U.S. Department of Energy	56	52	97,482	8,462
Gordon and Betty Moore Foundation	51	50	5,928	94
American Cancer Society	3	3	7,276	107
Children’s Tumor Foundation	1	1	759	630
American Parkinson Disease Association	0	0	181	12
Neurofibromatosis Therapeutic Acceleration Program	0	0	101	68
International Anesthesia Research Society	0	0	94	34
Australian National Data Service	0	0	92	67

Note that the fourth column reports the total number of outputs registered with Crossref and mentioning the given funder DOI, including grants, journal papers and all other record types.

It is interesting to compare the number of linked research outputs for a given funder with the total number of research outputs mentioning a given funder DOI. In general, for a funder that registers grants, the more research outputs mentioning this funder, the more links we should be able to find.

And for some funders (Japan Science and Technology Agency, Melanoma Research Alliance, Asia-Pacific Network for Global Change Research, Wellcome Trust, James S. McDonnell Foundation), the number of linked outputs is indeed high, as compared with how many outputs mention the funder in the first place. This suggests our procedure was quite successful in linking outputs funded by these funders, meaning that in general the metadata in their grants and the funding information in the research outputs match.

On the other hand, we have a few funders for which we managed to link only a very small fraction of research outputs. There are several potential explanations here. A simple one is that not all relevant grants have been deposited yet. For example, a funder might be registering new grants only, whereas many research outputs mention older, not yet registered grants. It is also possible that there are systematic differences in how the publishers deposit the funding information in articles and other outputs, and how it is given in grants. Such differences might prevent us from establishing links, contributing to the overall low percentage of linked grants.

The importance of being precise

Here are some examples of existing links that should’ve been found, but were not.

The award number in grant 10.48105/pc.gr.93156 is CTF-2020-01-004. This article: 10.3390/ijms22094716 mentions award number 2020‐01‐004 and the same funder (Children’s Tumor Foundation). It is very probable that this is the same grant, but our procedure expects exactly the same award number, and so the two were not linked.

Paper 10.1128/genomea.00159-18 contains award number 1931 and U.S. Department of Energy as the funder. There are two grants with the same award number and funder: 10.46936/10.25585/60001053 and 10.46936/genr.proj.2000.1931/60002530. It is difficult to choose between them, and these links were marked as unreliable.

These examples could be signs of systematic errors and/or discrepancies that effectively prevent linking of those funders’ grants.

What’s next

In problems such as linking grants to research outputs, there are typically two key ingredients of the success, which at the same time are the main areas of improvement: the quality of the metadata, and the strength of the linking approach.

The metadata could be improved greatly by addressing existing discrepancies between grants and research outputs and allowing (and encouraging!) the publishers to provide grant DOIs in the funding information. Thankfully, we are not alone in those efforts. Both this recent Upstream blog from Alexis-Michel Mugabushaka, and this Scholarly Kitchen post from Robert Harrington call for the development and adoption of grant DOIs in scholarly metadata.

In terms of the linking approach, there are some ideas that could be used to further improve the linking accuracy and completeness:

The verification by funder name could be fuzzy and allow for minor variations like typos or additional words.
Apart from replaced/replaced by and ancestor/descendant, there are other relationships between funders in the Funder Registry: continuation of, incorporates/incorporated into, merged with, renamed as, split into/split from. We could also consider those relationships during the funder validation.
Apart from the funder information, there is other information that could be potentially used for verification, for example, the names of the authors and the investigators, the domain, or keywords.

If you have any questions, do get in touch!

All numbers are as of March 8, 2022 ↩︎
https://api-crossref-org.turing.library.northwestern.edu/works?filter=award.number:gbmf7622 ↩︎
https://api-crossref-org.turing.library.northwestern.edu/works?filter=award.number:gbmf7622,type:grant ↩︎
https://api-crossref-org.turing.library.northwestern.edu/works?filter=award.number:2817 ↩︎
The code and data available here: https://gitlab.com/crossref/labs_data_analyses/-/tree/master/analyses/22-01-26-grants-matching ↩︎

Time to put the "R" back in "R&D"

Geoffrey Bilder — Mon, 07 Jun 2021 00:00:00 +0000

It is time to put the ‘R’ back into R&D.

The Crossref R&D team was originally created to focus on the kinds of research projects that have allowed Crossref to make transformational technology changes, launch innovative new services, and engage with entirely new constituencies. Some Illustrious projects that had their origins in the R&D group include:

DOI Content Negotiation
Similarity Check (originally CrossCheck)
ORCID (originally Author DOIs)
Crossmark
The Open Funder Registry
The Crossref REST API
Linked Clinical Trials
Event Data
Grant registration
ROR

And for each project that has graduated, there have been several that have not. Some projects were simply designed to gather data. Others just didn’t generate enough interest. You are not truly experimenting if you don’t fail occasionally too.

Recently we’ve been doing very little experimenting of any kind. Instead, the R&D team has mostly been seconded to the software development team to help them through a period of organisational and process change. We would not have made it through the past two years without their help.

But now we’re ready to focus on more ‘R’ and less ‘D’. And to that end, we are increasing the size of the team as well. Rachael Lammey will be joining the team as Head of Strategic Initiatives. She will work alongside our Principal R&D Developers, Esha Datta and Dominika Tkaczyk. Together they will be able to engage with new communities and immediately start experimenting with ways in which Crossref might be able to address their needs and use-cases.

We hope to soon add to our list of distinguished R&D project alumni.

Rationale & details

The Crossref R&D group (AKA “Labs”) has been the incubator of many services that are now in production and which form a fundamental part of Crossref’s identity and value. Similarity Check, ORCID, Crossmark, Open Funder Registry, The REST API, Linked Clinical Trials, and Event Data all started as R&D projects. More recently the enhancement of our reference matching infrastructure and the development and launch of ROR were also R&D projects.

And prior to the formation of the outreach group in 2015, the R&D group also led a critical function engaging with communities that, at the time, Crossref only had tangential connections with: PKP; DOAJ; funders; and the data and altmetrics communities.

But since the R&D group merged with the technology team back in 2019, we have done very little “R.” and very little community engagement of our own. Instead, the R&D team has supported the development team through a period of major cross-cutting projects and organisational change. Dominika has led the REST API rewrite and Esha—when she is not acting as technical lead on ROR—has also worked on the API rewrite and has kept Crossref metadata search on its feet. We would not have been able to make it through the past few years without their help.

Throughout this period, Rachael Lammey has continued the vital work of identifying, engaging with, and advocating for members of our community who we previously didn’t even know were members of our community.

The strength of the R&D group was that it combined outreach, product, and development functions. It was not only able to engage with new constituencies, but to quickly experiment with ways in which Crossref might be able to serve them. Previously, members of the R&D team would return from a conference or workshop that no Crossref member had ever attended before with a set of new contacts and ideas for new services and tools. They’d form interest groups and develop prototypes. Sometimes the interest groups would lead nowhere and sometimes the prototypes would be discarded. But critically, some of them would turn into the major services and organisations that now form a foundational part of open scholarly infrastructure.

And this is why it makes so much sense for Rachael to join the R&D team. The group is most effective when it is able to engage with new communities and immediately start experimenting with ways in which Crossref might be able to address their needs and use-cases. Rachael’s extensive experience in both product management and outreach—combined with Esha and Dominika’s experience leading development projects—is exactly what we need to reinvigorate the group and put the R back into R&D.

To kick off, we are going to be working on some small-ish, discrete projects. These include:

Better matching and linking of preprints to published articles;
Extending our journal title classification to cover all journal and conference proceedings titles; and
Tools to allow us to community-source structured metadata correction information and feed it back to our members.

We will consult with and update the community on the kinds of projects we are working on through regular tech updates and a revitalised Labs area of our website.

Oh- and we will certainly be designing some new Labs creatures.
–G

Double trouble with DOIs

Dominika Tkaczyk — Tue, 10 Mar 2020 00:00:00 +0000

Detective Matcher stopped abruptly behind the corner of a short building, praying that his loud heartbeat doesn’t give up his presence. This missing DOI case was unlike any other before, keeping him awake for many seconds already. It took a great effort and a good amount of help from his clever assistant Fuzzy Comparison to make sense of the sparse clues provided by Miss Unstructured Reference, an elegant young lady with a shy smile, who begged him to take up this case at any cost.

The final confrontation was about to happen, the detective could feel it, and his intuition rarely misled him in the past. He was observing DOI 10.2307/257306, which matched Miss Reference’s description very well. So far, there was no indication that DOI had any idea he was being observed. He was leaning on a wall across the street in a seemingly nonchalant way, just about to put out his cigarette. Empty dark streets and slowly falling snow together created an excellent opportunity to capture the fugitive.

Suddenly, Matcher heard a faint rustling sound. Out of nowhere, another shady figure, looking very much like 10.5465/amr.1982.4285592, appeared in front of the detective, crossed the street and started running away. Matcher couldn’t believe his eyes. These two DOIs had identical authors, year and title. They were even wearing identical volume and issue! He quickly noticed minor differences: slight alteration in the journal title and lack of the second page number in one of the DOIs, but this was likely just a random mutation. How could have he missed the other DOI? And more importantly, which of them was the one worried Miss Reference simply couldn’t live without?

TL;DR

Crossref metadata contains duplicates, i.e. items with different DOIs and identical (or almost identical) bibliographic metadata. This often happens when there is more than one DOI pointing to the same object. In some cases, but not all of them, one of the DOIs is explicitly marked as an alias of the other DOI.
In this blog post, I analyze those duplicates, that are not marked with an alias relation. The analysis shows that the problem exists, but is not big.
Among 524,496 DOIs tested in the analysis, 4,240 (0.8%) were flagged as having non-aliased duplicates. I divided those duplicates into two categories:
- Self-duplicate is a duplicate deposited by the same member as the other DOI, there were 3,603 (85%) of them.
- Other-duplicate is a duplicate deposited by a different member than the other DOI’s depositor, there were only 637 (15%) of them.
I used three member-level metrics to estimate the volume of duplicates deposited by a given member:
- Self-duplicate index is the fraction of self-duplicates in member’s DOIs: on average 0.67%.
- Other-duplicate index is the fraction of other-duplicates in a member’s DOIs: on average 0.13%.
- Global other-duplicate index is the fraction of globally detected other-duplicates involving a given member: on average 0.34%.

Introduction

In an ideal world, the relationship between research outputs and DOIs is one-to-one: every research output has exactly one DOI assigned and each DOI points to exactly one research output.

As we all know too well, we do not live in a perfect world, and this one-to-one relationship is also sometimes violated. One way to violate it is to assign more than one DOI to the same object. This can cause problems.

First of all, if there are two DOIs referring to the same object, eventually they both might end up in different systems and datasets. As a result, merging data between data sources becomes an issue, because we no longer can rely on comparing the DOI strings only.

Reference matching algorithms will also be confused when they encounter more than one DOI matching the input reference. They might end up assigning one DOI from the matching ones at random, or not assigning any DOI at all.

And finally, more than one DOI assigned to one object is hugely problematic for document-level metrics such as citation counts, and eventually affects h-indexes and impact factors. In practice, metrics are typically calculated per DOI, so when there are two DOIs pointing to one document, the citation count might be split between them, effectively lowering the count, and making every academic author’s biggest nightmare come true.

It seems we shouldn’t simply cover our eyes and pretend this problem does not exist. So what are we doing at Crossref to make the situation better?

It is possible for our members to explicitly mark a DOI as an alias of another DOI, if it was deposited by mistake. This does not remove the problem, but at least allows metadata consumers to access and use this information.
Whenever a DOI is registered or updated in Crossref, we automatically compare its metadata to the metadata of existing DOIs. If the metadata is too similar to the metadata of another DOI, this information is sent to the member and they have a chance to modify the metadata as they see fit.

Despite these efforts, we still see duplicates that are not explained by anything in the metadata. In this blog post, I will try to understand this problem better and assess how big it is. I also define three member-level metrics that can show how much a given member contributes to duplicates in the system and can flag members with unusually high fractions of duplicates.

Gathering the data

The data for this analysis was collected in the following way:
Only journal articles were considered in the analysis.
Only members with at least 5,000 journal article DOIs were considered in the analysis.
For each member, a random sample of 1,000 journal article DOIs was selected.
DOIs with no title, title shorter than 20 characters or shorter than 3 words were removed from each sample. This was done because items with short titles typically result in incorrectly flagged duplicates (false positives).
For each remaining DOI in the sample, a simple string representation was generated. This representation is a concatenation of the following fields: authors, title, container-title, volume, issue, page, published date.
This string representation was used as query.bibliographic in Crossref’s REST API and the resulting item list was examined.
If the original DOI came back as the first or the second hit, the relevance score difference between the first two hits is less than 1, they are both journal articles, and there is no relation (alias or otherwise) between them, the other one of the two is considered a duplicate of the original DOI. The score difference threshold was chosen through a manual examination of a number of cases. Most detected duplicates came back scored identically.

Overall results

In total, I tested 590 members and 524,496 DOIs. Among them, 4,240 DOIs (0.8%) were flagged as duplicates of other DOIs. This shows the problem exists, but is not huge.

I also analyzed separately two categories of duplicates:

self-duplicates are two DOIs with (almost) identical metadata, deposited by the same member,
other-duplicates are two DOIs with (almost) identical metadata, deposited by two different members.

Self-duplicates are more common: 3,603 (85%) of all detected duplicates are self-duplicates, and only 637 (15%) are other-duplicates. This is also good news: self-duplicates involve one member only, so they are easier to handle.

Self-duplicates

To explore the levels of self-duplicates among members, I used a custom member-level metric called self-duplicate index. Self-duplicate index is the fraction of self-duplicates among the member’s DOIs, in this case calculated over a sample.

On average, members have a very small self-duplicate index of 0.67%. In addition, in the samples of 44% of analyzed members no self-duplicates were found. The histogram shows the skewness of the distribution:

As we can see in the distribution, there are only a few members with high self-duplicate index. The table shows all members with the self-duplicate higher than 10%:

Name	Total DOIs	Sample size	Self-duplicate index
University of California Press	129,741	798	36%
Inderscience Publishers	127,729	998	29%
American Society of Hematology	137,124	990	24%
Pro Reitoria de Pesquisa, Pos Graduacao e Inovacao - UFF	7,756	919	19%
American Diabetes Association	49,536	946	18%

Other-duplicates

Other-duplicate index is the fraction of other duplicates among the member’s DOIs, in this case calculated from a sample.

On average, members have a very low other-duplicate index of only 0.13%. What is more, 89% members have no other-duplicates in the sample, and the distribution is even more skewed than in the case of self-duplicates:

Here is the list of all members with more than 2% of other-duplicates in the sample:

Name	Total DOIs	Sample size	Other-duplicate index
American Bryological and Lichenological Society	5,593	844	41%
Maney Publishing	15,342	832	6%
JSTOR	1,612,174	864	4%
American Mathematical Society (AMS)	83,015	844	4%

American Bryological and Lichenological Society is a clear outlier with 41% of their sample flagged as duplicates. Interestingly, all those duplicates come from one other member only (JSTOR) and JSTOR was the first to deposit them.

Similarly, all other-duplicates detected in the American Mathematical Society’s sample are shared with JSTOR, and JSTOR was the first to deposit them.

Maney Publishing’s 51 other-duplicates are all shared with a member not listed in this table: Informa UK Limited.

JSTOR is the only member in this table, whose 36 other-duplicates are shared with multiple (8) members.

Another interesting observation is that the members in this table (apart from JSTOR) are rather small or medium, in terms of total DOIs registered by them. It is also worrying that Informa UK Limited, a member that shares 51 other-duplicates flagged in Maney Publishing’s sample, was not flagged by this index. The reason might be differences in the overall number of registered DOIs: two members that deposited the same number of other-duplicates, but have different overall numbers of registered DOIs, will have different other-duplicate indexes.

To address this issue, I looked at a third index called global other-duplicate index. Global other-duplicate index is the fraction of globally detected other-duplicates involving a given member.

Global other-duplicate index has a useful interpretation: it tells us how much the overall number of other-duplicates would drop, if the given member resolved all its other-duplicates (for example by setting appropriate relations or correcting the metadata so that it is no longer so similar).

Here is the list of members with global-duplicate index higher than 2%:

Name	Total DOIs	Global other-duplicate index
JSTOR	1,612,174	69%
American Bryological and Lichenological Society	5,593	54%
Informa UK Limited	4,275,507	15%
Maney Publishing	15,342	8%
American Mathematical Society (AMS)	83,015	6%
Project Muse	326,300	5%
Wiley	8,003,815	3%
Elsevier BV	16,268,943	3%
Liverpool University Press	31,870	3%
Cambridge University Press (CUP)	1,621,713	2%
Ovid Technologies (Wolters Kluwer Health)	2,152,723	2%
University of Toronto Press Inc. (UTPress)	46,778	2%

Note that the values add up to more than 100%. This is because in every other-duplicate there are two members involved, so the involvement adds up to 200%.

As we can see, all the members from the previous table are in this one as well. Apart from them, however, this index flagged several large members. Among them, Informa UK Limited, that was missing from the previous table.

All the indexes defined here are useful in identifying members that contribute a lot of duplicates to the Crossref metadata. They can be used to help to clean up the metadata, and also to monitor the situation in the future.

Limitations

It is important to remember that index values presented here were calculated on a single sample of DOIs drawn for a given member. The values would be different if a different sample was used, and so they shouldn’t be treated as exact numbers.

The tables include members with the index exceeding a certain threshold, chosen arbitrarily, for illustrative purposes. Different runs with different samples could result in different members being included in the tables, especially in their lower parts.

To obtain more stable values of indexes, multiple samples could be used. Alternatively, in the case of smaller members, exact values could be calculated from all their DOIs.

What's your (citations') style?

Dominika Tkaczyk — Tue, 29 Oct 2019 00:00:00 +0000

Bibliographic references in scientific papers are the end result of a process typically composed of: finding the right document to cite, obtaining its metadata, and formatting the metadata using a specific citation style. This end result, however, does not preserve the information about the citation style used to generate it. Can the citation style be somehow guessed from the reference string only?

TL;DR

I built an automatic citation style classifier. It classifies a given bibliographic reference string into one of 17 citation styles or “unknown”.
The classifier is based on supervised machine learning. It uses TF-IDF feature representation and a simple Logistic Regression model.
For training and testing, I used datasets generated automatically from Crossref metadata.
The accuracy of the classifier estimated on the test set is 94.7%.
The classifier is open source and can be used as a Python library or REST API.

Introduction

Threadgill-Sowder, J. (1983). Question Placement in Mathematical Word Problems. School Science and Mathematics, 83(2), 107-111

This reference is the end result of a process that typically includes: finding the right document, obtaining its metadata, and formatting the metadata using a specific citation style. Sadly, the intermediate reference forms or the details of this process are not preserved in the end result. In general, just by looking at the reference string we cannot be sure which document it originates from, what its metadata is, or which citation style was used.

Global multi-billion dollar fashion industry proves without a doubt that people care about their fashion style. But why should we care about the citation style used to generate a specific reference? This might seem like an insignificant piece of information, but it can be a powerful clue when we try to solve tasks like:

Reference parsing, i.e., extracting metadata from the reference string. If the style is known, we also know where to expect metadata fields in the string, and it is typically enough to use simple regular expressions instead of complicated (and slow) machine learning-based parsers.
Discipline/topic classification. Citation styles used in documents correlate with their discipline. As a result, knowing the citation style used in the document could provide a useful clue for a discipline classifier.
Extracting references from documents. Conforming to a specific style might suggest that the reference string was correctly located within a larger document.

Even though the style is not directly mentioned in the reference string, the string contains useful clues. Some styles will abbreviate the authors’ first names, and others won’t. Some will place the year in parentheses, others separate it with commas. The presence of such fragments in the reference string can be used as the input for the style classifier.

I used these clues to build an automatic style classifier. It takes a single reference string on the input and classifies it into one of 17 styles or “unknown”. You can use it as a Python library or via REST API. The source code is also available. If you find this project useful, I would love to hear about it!

And if you are interested in more details about the classifier and how it was built, read on.

Data

The data for the experiments was generated automatically. The training and the test set were generated in the same way but from two different samples. The process was the following:

5,000 documents were randomly chosen from Crossref collection.
Each document was formatted into 17 citation styles. This resulted in 85,000 pairs (reference string, citation style).
Very short reference strings were removed. A short reference string typically results from very incomplete metadata of the document.
From a number of randomly selected references, I removed fragments like the name of the month. These fragments appear in the automatically generated reference strings because sometimes months are included in the metadata records in Crossref collection. However, they rarely appear in the real-life reference strings, so removing them made the dataset more reliable.
5,000 strings labelled as “unknown” were also added. These were generated by randomly swapping the words in the “real” reference strings.

This process resulted in two sets: training set containing 87,808 data points and test set containing 87,625 data points. The training set was used to choose various classification parameters and to train the final model. The test set was used to obtain the final estimation of the classifier’s accuracy.

Styles

The classifier was trained on the following 17 citation styles (+ “unknown”):

acm-sig-proceedings
american-chemical-society
american-chemical-society-with-titles
american-institute-of-physics
american-sociological-association
apa
bmc-bioinformatics
chicago-author-date
elsevier-without-titles
elsevier-with-titles
harvard3
ieee
iso690-author-date-en
modern-language-association
springer-basic-author-date
springer-lecture-notes-in-computer-science
vancouver

These 17 styles were chosen to cover a vast majority of references that we see in the real-life data, without including too many variants of very similar styles.

If you need a different style set, fear not. You can use the library to train your own model based on exactly the styles you need.

Features

Our learning algorithm cannot work directly with the raw text on the input. It needs numerical features. In the case of text classification (and reference strings are text), one very common feature representation is bag-of-words. In the simplest variant, each feature represents a single word, and the value of the feature is binary: 1 if the word is present in the text, 0 otherwise.

There are many variants of this representation, for example:

The input text typically undergoes normalization before the features are extracted. Depending on the use case, this might include lowercasing, removing punctuation, bringing the words to their canonical form by stemming, etc.
We do not have to use single words as features. In some use cases, it is beneficial to use n-grams, which correspond to fixed-length sequences of words.
Instead of binary values, we might want to use some other feature weight schemes, such as the famous TF-IDF representation.

Our use case is not a typical case of text classification. We cannot use raw words as features, as words do not carry the information about the citation style. Imagine the same document formatted in different styles –– those reference strings will contain the same words, and the learning algorithm won’t be able to distinguish between them.

As a side note, in some cases, some specific words might be important. For example, if the reference contains the word “algorithm”, chances are the document is from computer science. If so, then perhaps the citing paper is from computer science as well. And in computer science, some styles are more popular than others. Machine learning algorithms are pretty good at detecting such correlations in the data. In the first version of our classifier, however, we do not take this into account. This keeps things simpler.

If not words, then what matters in our case? It seems that the information about the style is present in punctuation, capitalization and abbreviations.

To capture these clues, before extracting the features we first map our reference string into a sequence of “word types” (or “character types”). The types are the following: lowercase-word, lowercase-letter, uppercase-word, uppercase-letter, capitalized-word, other-word, year, number, dot, comma, left-parenthesis, right-parenthesis, left-bracket, right-bracket, colon, semicolon, slash, dash, quote, other.

In addition, we mark the beginning and the end of the reference string with special types start and end.

So for example this string:

Eberlein, T. J. Yearbook of Surgery 2006, 322–324.

is mapped into this sequence:

start capitalized-word comma uppercase-letter dot uppercase-letter dot capitalized-word lowercase-word capitalized-word year comma number dash number dot end

This transformation effectively brings together different words, as long as their form is the same.

After transforming the reference string we extract 2-grams, 3-grams and 4-grams. The values of the features are TF-IDF weights.

Some example features in our representation include:

lowercase-word lowercase-word lowercase-word lowercase-word - a sequence of four lowercase words. It is most likely the part of the article title and won’t have a huge impact on the decision about the citation style.
capitalized-word comma uppercase-letter dot - typical representation of an author in some styles, where the first name is given as an initial only and follows the last name.
left-parenthesis year right-parenthesis - typical for styles that enclose the year in parentheses.
number dash number - this sequence is most likely pages range.

Learning algorithm

I tested four learning algorithms (naive Bayes, logistic regression, linear support vector classification and random forest) in a 5-fold cross validation on the training set. The plot shows the distribution of accuracies obtained by each algorithm:

Based on these results, logistic regression was chosen as the algorithm with the best mean accuracy and the lowest variance of the results.

Final accuracy estimation

The final model was trained on the entire training set and evaluated on the test set. As evaluation metric accuracy was used. In this case, accuracy is simply the fraction of the references in the test set correctly classified by the classifier.

The accuracy on the test set was 94.7%. The confusion matrix shows which styles were most often confused with each other:

The most often confused styles are chicago-author-date and american-sociological-association. Let’s see some example strings from these two styles:

Legros, F. 2003. "Can Dispersive Pressure Cause Inverse Grading in Grain Flows?: Reply." Journal of Sedimentary Research 73(2):335–335
Legros, F. 2003. "Can Dispersive Pressure Cause Inverse Grading in Grain Flows?: Reply." Journal of Sedimentary Research 73 (2) : 335–335

Clarke, Jennie T. 2011. "Recognizing and Managing Reticular Erythematous Mucinosis." Archives of Dermatology 147(6):715
Clarke, Jennie T. 2011. "Recognizing and Managing Reticular Erythematous Mucinosis." Archives of Dermatology 147 (6) : 715

Chalmers, Alan, and Richard Nicholas. 1983. "Galileo on the Dissipative Effect of a Rotating Earth." Studies in History and Philosophy of Science Part A 14(4):315–40
Chalmers, Alan, and Richard Nicholas. 1983. "Galileo on the Dissipative Effect of a Rotating Earth." Studies in History and Philosophy of Science Part A 14 (4) : 315–340

It seems that the styles are indeed very similar. The strings look almost identical, apart from spacing, which is not included in any way in our feature representation. No wonder that the classifier confuses these two styles a lot.

A more detailed analysis of the classifier can be found here.

What if I told you that bibliographic references can be structured?

Dominika Tkaczyk — Mon, 08 Jul 2019 00:00:00 +0000

Last year I spent several weeks studying how to automatically match unstructured references to DOIs (you can read about these experiments in my previous blog posts). But what about references that are not in the form of an unstructured string, but rather a structured collection of metadata fields? Are we matching them, and how? Let’s find out.

TL;DR

43% of open/limited references deposited with Crossref have no publisher-asserted DOI and no unstructured string. This means they need a matching approach suitable for structured references. [EDIT 6th June 2022 - all references are now open by default].
I adapted our new matching algorithms: Search-Based Matching (SBM) and Search-Based Matching with Validation (SMBV) to work with both structured and unstructured references.
I compared three matching algorithms: Crossref’s current (legacy) algorithm, SBM and SBMV, using a dataset of 2,000 structured references randomly chosen from Crossref’s references.
SBMV and the legacy algorithm performed almost the same. SBMV’s F1 was slightly better (0.9660 vs. 0.9593).
Similarly as in the case of unstructured references, SBMV achieved slightly lower precision and better recall than the legacy algorithm.

Introduction

Those of you who often read scientific papers are probably used to bibliographic references in the form of unstructured strings, as they appear in the bibliography, for example:

[5] Elizabeth Lundberg, “Humanism on Gallifrey,” Science Fiction Studies, vol. 40, no. 2, p. 382, 2013.

This form, however, is not the only way we can store the information about the referenced paper. An alternative is a structured, more machine-readable form, for example using BibTeX format:

@article{Elizabeth_Lundberg_2013,
year = 2013,
publisher = {{SF}-{TH}, Inc.},
volume = {40},
number = {2},
pages = {382},
author = {Elizabeth Lundberg},
title = {Humanism on Gallifrey},
journal = {Science Fiction Studies}
}

Probably the most concise way to provide the information about the referenced document is to use its identifier, for example (🥁drum roll…) the DOI:

<https://doi-org.turing.library.northwestern.edu/10.5621/sciefictstud.40.2.0382>

It is important to understand that these three representations (DOI, structured reference and unstructured reference) are not equivalent. The amount of information they carry varies:

The DOI, by definition, provides the full information about the referenced document, because it identifies it without a doubt. Even though the metadata and content are not directly present in the DOI string, they can be easily and deterministically accessed. It is by far the preferred representation of the referenced document.
The structured reference contains the metadata of the referenced object, but it doesn’t identify the referenced object without a doubt. In our example, we know that the paper was published in 2013 by Elizabeth Lundberg, but we might not know exactly which paper it is, especially if there are more than one document with the same or similar metadata.
The unstructured reference contains the metadata field values, but without the names of the fields. This also doesn’t identify the referenced document, and even its metadata is not known without a doubt. In our example, we know that the word “Science” appears somewhere in the metadata, but we don’t know for sure whether it is a part of the title, journal title, or maybe the author’s (very cool) name.

The diagram presents the relationships between all these three forms:

The arrows show actions that Crossref has to perform to transform one form to another.

Green transformations are in general easy and can be done without introducing any errors. The reason is that green arrows go from more information to less information. We all know how easy it is to forget important stuff!

Green transformations are typically performed when the publication is being created. At the beginning the author can access the DOI of the referenced document, because they know exactly which document it is. Then, they can extract the bibliographic metadata (the structured form) of the document based on the DOI, for example by following the DOI to the document’s webpage or retrieving the metadata from Crossref’s REST API. Finally, the structured form can be formatted into an unstructured string using, for example, the CiteProc tool.

We’ve also automated it further and these two green transformation (getting the document’s metadata based on the DOI and formatting it into a string) can be done in one go using Crossref’s content negotiation.

Red transformations are often done in systems that store bibliographic metadata (like our own metadata collection), often at a large scale. In these systems, we typically want to have DOIs (or other unique identifiers) of the referenced documents, but in practise we often have only structured and/or unstructured form. To fix this, we match references. Some systems also perform reference parsing (thankfully, we discovered we do not need to do this in our case).

In general, red transformations are difficult, because we have to go from less information to more information, effectively recreating the information that has been lost during paper writing. This requires a bit of reasoning, educated guessing, and juggling probabilities. Data errors, noise, and sparsity make the situation even more dire. As a result, we do not expect any matching or parsing algorithm to be always correct. Instead, we perform evaluations (like in this blog post) to capture how well they perform on average.

My previous blog post focused on matching unstructured references to DOIs (long red “matching” arrow). In this one, I analyse how well we can match structured references to DOIs (short red “matching” arrow).

References in Crossref

You might be asking yourself how important it is to have the matching algorithm working for both structured and unstructured references. Let’s look more closely at the references our matching algorithm has to deal with.

29% of open/limited references deposited with Crossref already have the DOI provided by the publisher member. At Crossref, when we come across those references, we start dancing on a rainbow to the tunes of Linkin Park, while the references holding their DOIs sprinkle from the sky. Some of us sing along. We live for those moments, so if you care about us, please provide as many DOIs in your references as possible!

You might be wondering how we are sure these publisher-provided DOIs are correct. The short answer is that we are not. After all, the publisher might have used an automated matcher to insert the DOIs before depositing the metadata. Nevertheless, our current workflow assumes these publisher-provided DOIs are correct and we simply accept them as they are.

Unfortunately, the remaining 71% of references are deposited without a DOI. Those are the references we try to match ourselves.

Here is the distribution of all the open/limited references:

17% of the references are deposited with no DOI and both structured and unstructured form. 11% have no DOI and only an unstructured form, and 43% have no DOI and only a structured form. These 43% cannot be directly processed by the unstructured matching algorithm.

This distribution clearly shows that we need a matching algorithm able to process both structured and unstructured references. If our algorithm worked only with one type, we would miss a large percentage of the input references, and the quality of our citation metadata would be questionable.

The analysis

Let’s get to the point. I evaluated and compared three matching algorithms, focusing on the structured references.

The first algorithm is one of the legacy algorithms currently used in Crossref. It uses fuzzy querying in a relational database to find the best matching DOI for the given structured reference. It can be accessed through a Crossref OpenURL query.

The second algorithm is an adaptation of the Search-Based Matching (SBM) algorithm for structured references. In this algorithm, we concatenate all metadata fields of the reference and use it to search in the Crossref’s REST API. The first hit is returned as the target DOI if its relevance score exceeds the predefined threshold.

The third algorithm is an adaptation of the Search-Based Matching with Validation (SBMV) for structured references. Similarly as in the case of SBM, we also concatenate all metadata fields of the input reference and use it to search in the Crossref’s REST API. Next, a number of top hits are considered as candidates and their similarity score with the input reference is calculated. The candidate with the highest similarity score is returned as the target DOI if its score exceeds the predefined threshold. The similarity score is based on fuzzy comparison of the metadata field values between the candidate and the input reference.

I compared these three algorithms on a test set composed of 2,000 structured bibliographic references randomly chosen from Crossref’s metadata. For each reference, I manually checked the output of all matching algorithms, and in some cases performed additional manual searching. This resulted in the true target DOI (or null) assigned to each reference.

The metrics are the same as in the previous evaluations: precision, recall and F1 calculated over the set of input references.

The thresholds for SBM and SBMV algorithms were chosen on a separate validation dataset. The validation dataset also contains 2,000 structured references with manually-verified target DOIs.

The results

The plot shows the results of the evaluation of all three algorithms:

The vertical black lines on top of the bars represent the confidence intervals.

As we can see, SBMV and the legacy approach achieved very similar results. SBMV slightly outperforms the legacy approach in F1: 0.9660 vs. 0.9593.

SBMV is slightly worse that the legacy approach in precision (0.9831 vs. 0.9929) and better in recall (0.9495 vs. 0.9280).

The SBM algorithm performs the worst, especially in precision. Why is there such a huge difference between SBM and SBMV? The algorithms differ in the post-processing validation stage. SBM relies on the ability of the search engine to select the best target DOI, while SBMV re-scores a number of candidates obtained from the search engine using custom similarity. The results here suggest that in the case of structured references, the right target DOI is usually somewhere close to the top of the search results, but often it is not in the first position. One of the reasons might be missing titles in 76% of the structured references, which can confuse the search engine.

Let’s look more closely at a few interesting cases in our test set:

first-page = 1000
article-title = Sequence capture using PCR-generated probes: a cost-effective method of targeted high-throughput sequencing for nonmodel organisms
volume = 14
author = Peñalba
year = 2014
journal-title = Molecular Ecology Resources

The reference above was successfully matched by SBMV to https://doi-org.turing.library.northwestern.edu/10.1111/1755-0998.12249, even though the document’s volume and pages are missing from Crossref’s metadata.

issue = 2
first-page = 101
volume = 6
author = Abraham
year = 1987
journal-title = Promoter: An Automated Promotion Evaluation System

Here the structure incorrectly labels article title as journal title. Despite this, the reference was correctly matched by our brave SBMV to https://doi-org.turing.library.northwestern.edu/10.1287/mksc.6.2.101.

author = Marshall Day C.
volume = 39
first-page = 572
year = 1949
journal-title = India. J. A. D. A.

Above we have most likely a parsing error. A part of the article title appears in the journal name, and the main journal name is abbreviated. ‘I see what you did there, my old friend Parsing Algorithm! Only a minor obstacle!’ said SBMV, and matched the reference to https://doi-org.turing.library.northwestern.edu/10.14219/jada.archive.1949.0114.

volume = 5
year = 2015
article-title = A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels
journal-title = BMJ Open

Here the the page number and author are not in the structure, but our invincible SBMV jumped over the holes left by the missing metadata and gracefully grabbed the right DOI https://doi-org.turing.library.northwestern.edu/10.1136/bmjopen-2015-009138.

issue = 2
first-page = 533
volume = 30
author = Uthman BM
year = 1989
journal-title = Epilepsia

In this case we have a mismatch in the page number (“533” vs. “S33”). But did SBMV give up and burst into tears? I think we already know the answer! Of course, it conquered the nasty typo with the sword made of fuzzy comparisons (yes, it’s a thing!) and brought us back the correct DOI https://doi-org.turing.library.northwestern.edu/10.1111/j.1528-1157.1989.tb05823.x.

Structured vs. unstructured

How does matching structured references compare to matching unstructured references?

The general trends are the same. For both structured and unstructured references, SBMV outperforms the legacy approach in F1, achieving worse precision and better recall. This tells us that our legacy algorithms are more strict and as a result they miss some links.

Structured reference matching seems easier than unstructured reference matching. The reason is that when we have the structure, we can compare the input reference to the candidate field by field, which is more precise than using the unstructured string.

Structured matching, however, in practise brings new challenges. One big problem is data sparsity. 15% of structured references without DOIs have fewer than four metadata fields. This is not always enough to identify the DOI. Also, 76% of the structured references without DOIs do not contain the article title, which poses a problem for candidate selection using the search engine.

What’s next?

So far, I have focused on evaluating SBMV for unstructured and structured references separately. 17% of the open/limited references at Crossref, however, have both unstructured and structured form. In those cases, it might be beneficial to use the information from both forms. I plan to perform some experiments on this soon.

The data and code for this evaluation can be found at https://github.com/CrossRef/reference-matching-evaluation. The Java version of SBMV (for both structured and unstructured references) can be found at https://gitlab.com/crossref/search-based-reference-matcher.

Reference matching: for real this time

Dominika Tkaczyk — Tue, 18 Dec 2018 00:00:00 +0000

In my previous blog post, Matchmaker, matchmaker, make me a match, I compared four approaches for reference matching. The comparison was done using a dataset composed of automatically-generated reference strings. Now it’s time for the matching algorithms to face the real enemy: the unstructured reference strings deposited with Crossref by some members. Are the matching algorithms ready for this challenge? Which algorithm will prove worthy of becoming the guardian of the mighty citation network? Buckle up and enjoy our second matching battle!

TL;DR

I evaluated and compared four reference matching approaches: the legacy approach based on reference parsing, and three variants of search-based matching.
The dataset comprises 2,000 unstructured reference strings from the Crossref metadata.
The metrics are precision and recall calculated over the citation links. I also use F1 as a standard single-number metric that combines precision and recall, weighing them equally.
The best variant of search-based matching outperforms the legacy approach in F1 (96.3% vs. 92.5%), with the precision worse by only 0.9% (98.09% vs. 98.95%), and the recall better by 8.9% (94.56% vs. 86.85%).
Common causes of SBMV’s errors are: incomplete/erroneous metadata of the target documents, and noise in the reference strings.
The results reported here generalize to the subset of references in Crossref that are deposited without the target DOI and are present in the form of unstructured strings.

Introduction

In reference matching, we try to find the DOI of the document referenced by a given input reference. The input reference can have a structured form (a collection of metadata fields) and/or an unstructured form (a string formatted in a certain citation style).

In my previous blog post, I used reference strings generated automatically to compare four matching algorithms: Crossref’s legacy approach based on reference parsing and three variations of search-based matching. The best algorithm turned out to be Search-Based Matching with Validation (SBMV). SBMV uses our REST API’s bibliographic search function to select the candidate target documents, and a separate validation-scoring procedure to choose the final target document. The legacy approach and SBMV achieved very similar average precision, and SBMV was much better in average recall.

This comparison had important limitations, which affect the interpretation of these results.

First of all, the reference strings in the dataset might be too perfect. Since they were generated automatically from the Crossref metadata records, any piece of information present in the string, such as the title or the name of the author, will exactly match the information in Crossref’s metadata. In such a case, a matcher comparing the string against the record can simply apply exact matching and everything should be fine.

In real life, however, we should expect all sorts of errors and noise in the reference strings. For example, a string might have been manually typed by a human, so it can have typos. The string might have been scraped from the PDF file, in which case it could have unusual unicode characters, ligatures or missing and extra spaces. A string can also have typical OCR errors, if it was extracted from a scan.

These problems are typical for messy real-life data, and our matching algorithms should be robust enough to handle them. However, when we evaluate and compare approaches using the perfect reference strings, the results won’t tell us how well the algorithms handle harder, noisy cases. After all, even if you repeatedly win chess games against your father, it doesn’t mean you will likely defeat Garry Kasparov (unless, of course, you are Garry Kasparov’s child, in which case, please pass on our regards to your dad!).

Even though I attempted to make the data more similar to the noisy real-life data by simulating some of the possible errors (typos, missing/extra spaces) in two styles, this might not be enough. We simply don’t know the typical distribution of the errors, or even what all the possible errors are, so our data was probably still far from the real, noisy reference strings.

The differences in the distributions are a second major issue with the previous experiment. To build the dataset, I used a random sample from Crossref metadata, so the distribution of the cited item types (journal paper, conference proceeding, book chapter, etc.) reflects the overall distribution in our collection. However, the distribution in real life might be different if, for example, journal papers are on average cited more often than conference proceedings.

Similarly, the distribution of the citation styles is most likely different. To generate the reference strings, I used 11 styles distributed uniformly, while the real distribution most likely contains more styles and is skewed.

All these issues can be summarized as: the data used in my previous experiment is different from the data our matching algorithms have to deal with in the production system. Why is this important? Because in such a case, the evaluation results do not reflect the real performance in our system, just like the child’s score on the math exam says nothing about their score on the history test. We can hope my previous results accurately showed the strengths and weaknesses of each algorithm, but the estimations could be far off.

So, can we do better? Sure!

This time, instead of automatically-generated reference strings, I will use real reference strings found in the Crossref metadata. This will give us a much better picture of the matching algorithms and their real-life performance.

Evaluation

This time the evaluation dataset is composed of 2,000 unstructured reference strings from the Crossref metadata, along with the target true DOIs. The dataset was prepared mostly manually:

First, I drew a random sample of 100,000 metadata records from the system.
Second, I iterated over all sampled items, and extracted those unstructured reference strings, that do not have the DOI provided by the member.
Next, I randomly sampled 2,000 reference strings.
Finally, I assigned a target DOI (or null) to each reference string. This was done by verifying DOIs returned by the algorithms and/or manual searching.

The metrics this time are based on the citation links. A citation link points from the reference (or the document containing the reference) to the referenced (target) document.

When we apply a matching algorithm to a set of reference strings in our collection, we get a set of citation links between our documents. I will call those citation links returned links.

On the other hand, in our collection we have real, true links between the documents. In the best-case scenario, the set of true links and the set of returned links are identical. But we don’t live in a perfect world and our matching algorithms make mistakes.

To measure how close the returned links are to the true links, I used precision, recall and F1. This time they are calculated over all citation links in the dataset. More specifically:

Precision is the fraction of the returned links that are correct. Precision answers the question: if I see a citation link A->B in the output of a matcher, how certain can I be that paper A actually cites paper B?
Recall is the percentage of true links that were returned by the algorithm. Recall answers the question: if paper A cites paper B and B is in the collection, how certain can I be that the matcher’s output contains the citation link A->B?
F1 is the harmonic mean of precision and recall.

In the previous experiment, I also used precision, recall and F1, but they were calculated for each target document and then averaged. This time precision, recall and F1 are not averaged but simply calculated over all citation links. This is a more natural approach, since now the dataset comprises isolated reference strings rather than target documents, and in practice each target document has at most one incoming reference.

I tested the same four approaches as before:

the legacy approach, based on reference parsing
SBM with a simple threshold, which searches for the reference string in the search engine and returns the first hit, if its relevance score exceeds the predefined threshold
SBM with a normalized threshold, which searches for the reference string in the search engine and returns the first hit, if its relevance score divided by the string length exceeds the predefined threshold
SBMV, which first applies SBM with a normalized threshold to select a number of candidate items, and a separate validation procedure is used to select the final target item

All the thresholds are parameters which have to be set prior to the matching. The thresholds used in the experiments were chosen using a separate dataset, as the values maximizing the F1 of each algorithm.

Results

The plot shows the overall results of all tested approaches:

The exact values are also given in the table (the best result for each metric is bolded):

	precision	recall	F1
legacy approach	0.9895	0.8685	0.9251
SBM (simple threshold)	0.8686	0.8191	0.8431
SBM (normalized threshold)	0.7712	0.9121	0.8358
SBMV	0.9809	0.9456	0.9629

As we can see, the legacy approach is the best in precision, slightly outperforming SBMV. In recall, SBMV is clearly the best, which also decided about its victory over the legacy approach in F1.

How do these results compare to the results from my previous blog post? The overall trends (the legacy approach slightly outperforms SBMV in precision, and SBMV outperforms the legacy approach in recall and F1) are the same. The most important differences are: 1) on the real dataset SBM without validation is worse than the legacy approach, and 2) this time the algorithms achieved much higher recall. These differences are most likely related to the difference in data distributions explained before.

SBMV’s strengths and weaknesses

Let’s look at a few example cases where SBMV successfully returned the correct DOI, while the legacy approach failed.

Lundqvist D, Flykt A, Ohman A: The Karolinska Directed Emotional Faces - KDEF, CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet. 1998

matched to https://doi-org.turing.library.northwestern.edu/10.1037/t27732-000

The target item is a dataset, which means unusual metadata fields and an unusual reference string.

Schminck, A. , ‘The Beginnings and Origins of the “Macedonian” Dynasty’ in J. Burke and R. Scott , eds., Byzantine Macedonia: Identity, Image and History (Melbourne, 2000), 61–8.

matched to https://doi-org.turing.library.northwestern.edu/10.1163/9789004344730_006

This is an example of a book chapter. The reference string contains special quotes and dash characters.

R. Schneider,On the Aleksandrov-Fenchel inequality, inDiscrete Geometry and Convexity (J. E. Goodman, E. Lutwak, J. Malkevitch and R. Pollack, eds.), Annals of the New York Academy of Sciences440 (1985), 132–141.

matched to https://doi-org.turing.library.northwestern.edu/10.1111/j.1749-6632.1985.tb14547.x

In this case, spaces are missing in the reference string, which might be problematic for the parsing.

R. B. Husar andE. M. Sparrow, Int. J. Heat Mass Transfer11, 1206 (1968).

matched to https://doi-org.turing.library.northwestern.edu/10.1016/0017-9310(68)90036-7

This is another example of a reference string with missing spaces.

F. Cappello, A. Geist, W. Gropp, S. Kale, B. Kramer, and M. Snir. Toward exascale resilience: 2014 update. Supercomputing frontiers and innovations, 1(1), 2014.

matched to https://doi-org.turing.library.northwestern.edu/10.14529/jsfi140101

In this case authors are missing in the Crossref metadata.

Li KZ, Shen XT, Li HJ, Zhang SY, Feng T, Zhang LL. Ablation of the Carbon/carbon Composite Nozzle-throats in a Small Solid Rocket Motor[J]. Carbon, 2011, 49: 1 208–1 215

matched to https://doi-org.turing.library.northwestern.edu/10.1016/j.carbon.2010.11.037

Here we have unexpected spaces inside page numbers.

N. Kaloper, A. Lawrence and L. Sorbo, An Ignoble Approach to Large Field Inflation, JCAP 03 (2011) 023 [ arXiv:1101.0026 ] [ INSPIRE ].

matched to https://doi-org.turing.library.northwestern.edu/10.1088/1475-7516/2011/03/023

In this case we have an acronym of the journal name and additional arXiv id.

KrönerE. ?Stress space and strain space continuum mechanics?, Phys. Stat. Sol. (b), 144 (1987) 39?44.

matched to https://doi-org.turing.library.northwestern.edu/10.1002/pssb.2221440104

This reference string has a missing space, a missing word in the title, and incorrectly encoded special characters.

Suyemoto K. L., (1998) The functions of self-mutilationClinical Psychology Review 18(5): 531–554

matched to https://doi-org.turing.library.northwestern.edu/10.1016/s0272-7358(97)00105-0

In this case the space is missing between the title and the journal name.

Ono , N. 2011 Stable and fast update rules for independent vector analysis based on auxiliary function technique Proceedings of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics 189 192

matched to https://doi-org.turing.library.northwestern.edu/10.1109/aspaa.2011.6082320

The parsing can also have problems with missing punctuation, like in this case.

Hybertsen M.S., Witzigmann B., Alam M.A., Smith R.K. (2002) 1 113

matched to https://doi-org.turing.library.northwestern.edu/10.1023/a:1020732215449

In this case both title and journal name are missing from the reference string.

We can see from these examples that SBMV is fairly robust and able to deal with a small amount of noise in the metadata and reference strings.

What about the errors SBMV made? From the perspective of citation links, we have two types of errors:

False positives: incorrect links returned by the algorithm.
False negatives: links that should have been returned but weren’t.

When we apply SBMV instead of the legacy approach, the fraction of false positives within the returned links increases from 1.05% to 1.91%, and the fraction of false negatives within the true links decreases from 13.15% to 5.44%. This means with SBMV:

1.91% of the links in the algorithm’s output are incorrect
5.44% of the true links are not returned by the algorithm

We can also classify all the references in the dataset into several categories, based on the values of true and returned DOIs:

We have the following categories:

References matched to correct DOIs (1129 cases, returned and true blue)
References correctly not matched to anything (791 cases, returned and true white)
References not matched to anything, when they should be (58 cases, returned white, true grey)
References matched to wrong DOIs (7 cases, returned red, true yellow)
References matched to something, when they shouldn’t be matched to anything (15 cases, returned black, true white)

Note that in terms of these categories, precision is equal to:

And recall is equal to:

What are the most common causes of SBMV’s errors?

Incomplete or incorrect Crossref metadata. Even a perfect reference string formatted in the most popular citation style will not be matched, if the target record in the Crossref collection has many missing or incorrect fields.
Similarly, missing or incorrect information in the reference string is very problematic for the matchers.
Errors/noise in the reference string, such as:
- HTML/XML markup not stripped from the string
- multiple references mixed in one string
- spacing issues and typos
In a few cases a document related to the real target was matched, such as the book instead of its chapter, or the conference proceedings paper instead of the thesis.

Limitations

The most important limitation is the size of the dataset. Every item had to be verified manually, which significantly limited the possibility of creating a large set and also using a lot of independent sets.

Finally, the numbers reported here still don’t reflect the overall precision and recall of the current links in the Crossref metadata. This is because:

we still use the legacy approach for matching,
some references are deposited along with the target DOIs and are not matched by Crossref, these links are not analyzed here, and
in Crossref we have both unstructured and structured references, and in this experiment only the unstructured ones were tested.

What’s next?

The next experiment will be related to the structured references. Similarly as here, I will try to estimate the performance of the search-based matching approach and compare it to the performance of the legacy approach.

The evaluation framework, evaluation data and experiments related to the reference matching are available in the repository https://github.com/CrossRef/reference-matching-evaluation. Future experiments will be added there as well.

https://github.com/CrossRef/reference-matching-evaluation also contains the Python implementation of the SBMV algorithm. The Java implementation of SBMV is available in the repository https://gitlab.com/crossref/search_based_reference_matcher.

Matchmaker, matchmaker, make me a match

Dominika Tkaczyk — Mon, 12 Nov 2018 00:00:00 +0000

Matching (or resolving) bibliographic references to target records in the collection is a crucial algorithm in the Crossref ecosystem. Automatic reference matching lets us discover citation relations in large document collections, calculate citation counts, H-indexes, impact factors, etc. At Crossref, we currently use a matching approach based on reference string parsing. Some time ago we realized there is a much simpler approach. And now it is finally battle time: which of the two approaches is better?

TL;DR

I evaluated and compared four approaches to reference matching: the legacy approach based on reference parsing, and three variants of the new idea called search-based matching.
A large automatically generated dataset was used for the experiments. It is composed of 7,374 metadata records from the Crossref collection, each of which was formatted automatically into reference strings using 11 citation styles.
The main metrics used for the evaluation are precision and recall. I also use F1 as a standard metric that combines precision and recall into a single number, weighing them equally. All values are calculated for each metadata record separately and averaged over the dataset.
In general, search-based matching is better than the legacy approach in F1 and recall, but worse in precision.
The best variant of search-based matching outperforms the legacy approach in average F1 (84.5% vs. 52.9%), with the average precision worse by only 0.1% (99.2% vs 99.3%), and the average recall better by 88% (79.0% vs. 42.0%).
The best variant of search-based matching also outperforms the legacy approach in average F1 for each one of the 11 styles.
A weak spot of the parsing-based approach is degraded/noisy reference strings, which do not appear to use any of the known citation styles.
A weak spot of search-based approach is short reference strings, and in particular citation styles that do not include the title in the reference string.

Introduction

In reference matching, on the input we have a bibliographic reference. It can have the form of an unstructured string, such as:

(1) Adamo, S. H.; Cain, M. S.; Mitroff, S. R. Psychological Science 2013, 24, 2569–2574.

The input can also have the form of a structured reference, such as (BibTex format):

 @article{adamo2013,
 author = {Stephen H. Adamo and Matthew S. Cain and Stephen R. Mitroff},
 title = {Self-Induced Attentional Blink: A Cause of Errors in Multiple-Target Search},
 journal = {Psychological Science},
 volume = {24},
 number = {12},
 pages = {2569-2574},
 year = {2013}
}

The goal of matching is to find the document, which the input reference points to.

Matching algorithms

Matching references is not a trivial task even for a human, not to mention the machines, which are still a bit less intelligent than us (or so they want us to believe…). A typical meta-approach to reference matching might be to score the similarity between the input reference and the candidate target documents. The document most similar to the input is then returned as the target.

Of course, still a lot can go wrong here. We can have more than one potential target record with the same score (which one do we choose?). We can have only documents with low to medium scores (is the actual target even present in our collection?). We can also have errors in the input string (are the similarity scores robust enough?). Life’s tough!

The main difference between various matching algorithms is in fact how the similarity is calculated. For example, one idea might be to compare the records field by field (how similar is the title/author/journal in the input reference to the title/author/journal of our candidate target record?). This is roughly how the matching works currently at Crossref.

The main problem with this approach is that it requires a structured reference, and in practise, often all we have is a plain reference string. In such a case we need to extract the metadata fields from the reference string (this is called parsing). Parsing introduces errors, since no parser is omniscient. The errors propagate further and affect the scoring… you get the picture.

Luckily, as we have known for some time now, this is not the only approach. Instead of comparing structured objects, we could calculate the similarity between them using their unstructured textual form. This effectively eliminates the need for parsing, since the unstructured form is either already available on the input or can be easily generated from the structured form.

What about the similarity scores? We already know a powerful method for scoring the similarities between texts. Those are (you guessed it!) scoring algorithms used by search engines. Most of them, including Crossref’s, do not need a structured representation of the object, they are perfectly happy with just a plain text query.

So all we need to do is to pass the original reference string (or some concatenation of the reference fields, if only a structured reference is available) to the search engine and let it score the similarity for us. It will also conveniently sort the results so that it is easy to find the top hit.

Evaluation

So far so good. But which strategy is better? Is it better to develop an accurate parser, or just rely on the search engine? I don’t feel like guessing. Let’s try to answer this using (data) science. But first, we need to decompose our question into smaller pieces.

Question 1. How can I measure the quality of a reference matcher?

Generally speaking, this can be done by checking the resulting citation links. Simply put, the better the links, the better the matching approach must have been.

A few standard metrics can be applied here, including accuracy, precision, recall and F1. We decided to calculate precision, recall and F1 separately for each document in the dataset, and then average those numbers over the entire dataset.

When I say “documents”, I really mean “target documents”:

precision for a document X tells us, what percentage of links to X in the system are correct
recall for a document X tells us, what percentage of true links to X are present in the system
F1 is the harmonic mean of precision and recall

F1 is a single-number metric combining precision and recall. In F1 precision and recall are weighted equally. It is also possible to combine precision and recall using different weights, to place more emphasis on one of those metrics.

We decided to look at links from the target document’s perspective, because this is what the academic world cares about (i.e. how accurate the citation counts of academic papers are).

Calculating separate numbers for individual documents and averaging them within a dataset is the best way to have reliable confidence intervals (which makes the whole analysis look much smarter!).

Question 2. Which approaches should be compared?

In total we tested four reference matching approaches.

The first approach, called the legacy approach, is the approach currently used in Crossref ecosystem. It uses a parser and matches the extracted metadata fields against the records in the collection.

The second approach is the search-based matching (SBM) with a simple threshold. It queries the search engine using the reference string and returns the top hit from the results, if its relevance score exceeds the threshold.

The third approach is the search-based matching (SBM) with a normalized threshold. Similarly as in the simplest SBM, in this approach we query the search engine using the reference string. In this case the first hit is returned if its normalized score (the score divided by the reference length) exceeds the threshold.

Finally, the fourth approach is a variation of the search based matching, called search-based matching with validation (SBMV). In this algorithm we use additional validation procedure on top of SBM. First, SBM with a normalized threshold is applied and the search results with the scores exceeding the normalized threshold are selected as candidate target documents. Second, we calculate validation similarity between the input string and each of the candidates. This validation similarity is based on the presence of the candidate record’s metadata fields (year, volume, issue, pages, the last name of the first author, etc.) in the input reference string, as well as the relevance score returned by the search engine. Finally, the most similar candidate is returned as the final target document, if its validation similarity exceeds the validation threshold.

By adding the validation stage to the search-based matching we make sure that the same bibliographic numbers (year, volume, etc.) are present in both the input reference and the returned document. We also don’t simply take the first result, but rather use this validation similarity to choose from results scored similarly by the search engine.

All the thresholds are parameters which have to be set prior to the matching. The thresholds used in these experiments were chosen using a separate dataset, as the values maximizing the F1 of each algorithm.

Question 3. How to create the dataset?

Results

We could try to calculate our metrics for every single document in the system. Since we currently have over 100M of them, this would take a while, and we already felt impatient…

A faster strategy was to use sampling with all the tools statistics was so generous to provide. And this is exactly what we did. We used a random sample of 2500 items from our system, which is big enough to give reliable results and, as we will see later, produces quite narrow confidence intervals.

Apart from the sample, we needed some input reference strings. We generated those automatically by formatting the metadata of the chosen items using various citation styles. (Similarly to what happens when you automatically format the bibliography section for your article. Or at least we hope you don’t produce those reference strings manually…)

For each record in our sample, we generated 11 citation strings, using the following styles:

Well known citation styles from various disciplines:
- american-chemical-society (acs)
- american-institute-of-physics (aip)
- elsevier-without-titles (ewt)
- apa
- chicago-author-date
- modern-language-association (mla)
Known styles + random noise. To simulate not-so-clean data, we randomly added noise (additional spaces, deleted spaces, typos) to the generated strings of the following styles:
- american-institute-of-physics
- apa
Custom degraded “styles”:
- degraded: a simple concatenation of authors’ names, title, container title, year, volume, issue and pages
- one author: a simple concatenation of the first author’s name, title, container title, year, volume, issue and pages
- title scrambled: same as degraded, but with title words randomly shuffled

Some styles include the DOI in the reference string. In such cases we stripped the DOI from the string, to make the matching problem non-trivial.

An ideal matching algorithm will match every generated string to the record it was generated from. In practise, some of the expected matches will be missing, which will lower the recall of the tested matching approach. On the other hand, it is very probable that we will get the precision of 100%. To have the precision lower than 100%, we would have to have some unexpected matches to our sampled documents, which is unlikely. This is obviously not great, because we are missing a very important piece of information.

What can we do to “encourage” such mismatches to our sampled documents? We could generate additional reference strings of documents that are not in our sample, but are similar to the documents in our sample. Hopefully, we will see some incorrect links from those similar strings to our sampled documents.

For each sampled document I added up to 2 similar documents (I used, surprise surprise, our search engine to find the most similar documents). I ended up with 7,374 items in total (2,500 originally sampled and 4,874 similar items). For each item, 11 different reference strings were generated. Each reference string was then matched using the tested approaches and I could finally look at some results.

Results

First, let’s compare the overall results averaged over the entire dataset:

The small vertical black lines at the top of the boxes show the confidence intervals at the confidence level 95%. The table gives the exact values and the same confidence intervals. The best result for each metric is bolded.

	average precision	average recall	average F1
legacy approach	0.9933 (0.9910 - 0.9956)	0.4203 (0.4095 - 0.4312)	0.5289 (0.5164 - 0.5413)
SBM (simple threshold)	0.9890 (0.9863 - 0.9917)	0.7127 (0.7021 - 0.7233)	0.7866 (0.7763 - 0.7968)
SBM (normalized threshold)	0.9872 (0.9844 - 0.9901)	0.7905 (0.7796 - 0.8015)	0.8354 (0.8249 - 0.8458)
SBMV	0.9923 (0.9902 - 0.9945)	0.7902 (0.7802 - 0.8002)	0.8448 (0.8352 - 0.8544)

The confidence intervals given in the table are the ranges, in which it is 95% likely to have the real average precision, recall and F1. For example, we are 95% sure that the real F1 for SBMV in our entire collection is within the range 0.8352 - 0.8544.

As we can see, each metric has a different winner.

The legacy approach is the best in precision. This suggests the legacy approach is quite conservative and outputs a match only if it is very sure about it. This might also result in missing a number of true matches (false negatives).

According to the paired Student’s t-test, the difference between the average precision of the legacy approach and the average precision of the second best SBMV is not statistically significant. This means we cannot rule out that this difference is simply the effect of the randomness in sampling, and not the sign of the true difference.

SBM with a normalized threshold is the best in recall. This suggests that it is fairly tolerant and returns a lot of matches, which might also result in returning more incorrect matches (false positives). Also in this case the difference between the winner and the second best (SBMV) is not statistically significant.

SBMV is the best in F1. This shows that this approach balances precision and recall the best, despite being only the second best in both of those metrics. According to the paired Student’s t-test, the difference between SBMV and the second best approach (SBM with a normalized threshold) is statistically significant.

All variants of the search-based matching outperform the parsing-based approach in terms of F1, with statistically significant differences. This shows that in search based-matching it is possible to keep precision almost as good as in the legacy approach, and still include many more true positives.

Let’s also look at the same results split by the citation style:

For all styles the precision values are very high, and the legacy approach is slightly better than all variations of the search-based approach.

In terms of recall and F1 SBM with a simple threshold is better than the legacy approach in 8 out of 11 styles. The three styles for which the legacy approach outperforms SBM with a simple threshold are styles that do not include the title in the reference strings (acs, aip and ewt). The reason for this is that the simple threshold cannot be well calibrated for shorter and longer reference strings at the same time.

SBM with a normalized threshold and SBMV is better than the legacy approach in recall and F1 for all 11 styles.

The weak spot of the legacy approach is degraded and noisy reference strings, which do not appear to use any of the known citation styles.

The weak spot of the search-based matching is short reference strings, and in particular citation styles that do not include the title in the string.

Limitations

The limitations are related mostly to the method of building the dataset.

All the numbers reported here are estimates, since they were calculated on a sample.
The numbers show strengths and weaknesses of each approach, but they do not reflect the real precision and recall in the system:
- Since we included only 2 similar documents for each document in the sample, precision is most likely lower in the real data.
- We used a number of styles distributed uniformly. Of course in the real system the styles and their distribution might be different, which affects all the calculated numbers.

What does the sample say?

Dominika Tkaczyk — Fri, 09 Nov 2018 00:00:00 +0000

At Crossref Labs, we often come across interesting research questions and try to answer them by analyzing our data. Depending on the nature of the experiment, processing over 100M records might be time-consuming or even impossible. In those dark moments we turn to sampling and statistical tools. But what can we infer from only a sample of the data?

Imagine you are cooking soup. You just put some salt in it and now you are wondering if it is salty enough. What do you do next?

Option #1: Since you carefully measured 1/7 of a teaspoon of salt per 0.13 litres of soup (as always), you already know the soup is fine. Everyone else better stop asking silly questions and eat their soup.
Option #2: You stir everything carefully and taste a tablespoon. If it is not salty enough, you put more salt in the soup and repeat the tasting procedure.
Option #3: You eat a tablespoon of soup and it tastes fine. But wait, there’s more soup in the pot, what if the sip you’ve just tasted was somehow different than the rest? You decide it’s better to eat another spoon of soup (which tastes fine). Still, a lot of soup left, who knows what that tastes like? It might be safer to eat an entire bowl of soup. Hmm, still not sure, you’ve eaten such a small fraction of the soup, who can guarantee the rest tastes the same? You have no choice but to eat another bowl, and then some more… Ooops, now you have eaten the entire pot of soup! At least you can be 100% sure now that the soup was indeed salty enough. The problem is, there is no soup left, and also, you don’t feel so good. But people are getting hungry, so you start cooking a new batch…

If your answer was option #3, read on. Your life is going to get easier!

TL;DR

Sampling and confidence intervals can be used to estimate the mean of a certain feature, or the proportion of items passing a certain test, by calculating it only for a random sample of items, instead of the entire large set of items. Note that estimating =/= guessing.
Confidence intervals are a way of controlling the amount of uncertainty related to randomness in sampling.
The confidence interval has a form (estimated value - something, estimated value + something). Confidence interval at the confidence level 95% is interpreted as follows: we are 95% sure that the real value that we are estimating is within our calculated confidence interval.
The higher the confidence level (i.e. the more certain we want to be about the interval), the wider the interval has to be.
The larger the sample, the narrower the confidence interval.
We are never 100% sure that the value we are estimating is actually within our calculated confidence interval. By setting the confidence level high, we only make sure this is a very likely event.

The problem

Sampling and estimating drew my attention while I was working on the evaluation of the reference matching algorithms. In Crossref’s case, reference matching is the task of finding the target document DOI for the given input reference string, such as:

(1) Adamo, S. H.; Cain, M. S.; Mitroff, S. R. Psychological Science 2013, 24, 2569–2574.

Accurate reference matching is very important for the scientific community. Thanks to automatic reference matching we are able to find citing relations in large document sets, calculate citation counts, H-indexes, impact factors, etc.

For several weeks now I have been investigating a simple reference matching algorithm based on the search engine. In this algorithm, we use the input reference string as the query in the search engine, and we return the first item from the results as the target document. Luckily, at Crossref we already have a good search engine in place, so all the pieces are there.

I was interested in how well this simple algorithm works, i.e. how often the correct target document is found. For example, let’s say we have a reference string in APA citation style generated for a specific record in Crossref system. How certain can I be that it will be correctly matched to the record’s DOI?

I could calculate this directly by generating the APA reference string for every record in the system and trying to match those strings to DOIs. Since we already have over 100M records, this would take a while and I was getting impatient. So instead of eating the whole pot of soup, I decided to stir and taste just a little bit of it, or, academically speaking, use sampling and confidence intervals.

These statistical tools are useful in situations, where we have a large set of items, and we want to know the average of a certain feature of an item in our set, or the proportion of items passing a certain test, but calculating it directly is impossible or difficult. For example, we might want to know the average height of all women living in USA, the average salary of a Java programmer in London, or the proportion of book records in the Crossref collection. The entire set we are interested in is called a population and the value we are interested in is called a population average or a population proportion. Sampling and confidence intervals let us estimate the population average or proportion using only a sample of items, in a reliable and controlled way.

Experiments

In general I wanted to see, how well I can estimate the population proportion of records passing a certain test, using only a sample.

In the following experiments, the population is 1 million metadata records from the Crossref collection. I didn’t use the entire collection as the population, because I wanted to be able to calculate the real proportion and compare it to the estimates.

The test for a single record is: whether the APA reference string generated from said record is correctly matched to the record’s original DOI. In other words: if I generate the APA reference string from my record and use it as the query in Crossref’s search, will the record be the first element in the result list? Note that this proportion can also be interpreted as the probability that the APA reference string will be correctly matched to the target DOI.

Estimating from a sample

I took a random sample of size 100 from my population and calculated the proportion of the records correctly matched - this is called a sample proportion. In my case, the sample proportion is 0.92. This means that in my sample 92 reference strings were successfully matched to the right DOIs. Not too bad.

I could now treat this number as the estimate and assume that 0.92 is close to the population proportion. On the other hand, this is only a sample, and a rather small one, which raises doubts. What if our 92 correct matches happen to be the only correct matches in the entire 1M population? In such a case, our estimate of 0.92 would be very far from the population proportion. This uncertainty related to sampling randomness can be captured by the confidence interval.

Confidence interval

The confidence interval for my 100-point sample, at the confidence level 95%, is 0.8668-0.9732. This is interpreted as follows: we are 95% sure that the real population proportion is within the range 0.8668-0.9732. Note that the sample average (0.92) is exactly in the middle of this range.

100 items is not a big sample. Let’s calculate the confidence interval for a sample 10 times larger. From a sample of size 1000 I got the estimate 0.932, and the confidence interval 0.9164-0.9476. Based on this, we can be 95% sure that the real population proportion is within the range 0.9164-0.9476.

It seems the our interval got smaller when we increased the sample size. Let’s plot the intervals for a variety of sample sizes:

The blue line represents the estimated proportion for samples of different sizes, and the grey vertical lines are confidence intervals. The estimated proportion varies, because for each size a different sample was drawn.

We can see that increasing the sample size decreases the interval. This should make intuitive sense: if we have more data to estimate from, we can expect our estimate to be more reliable (i.e. closer to the population proportion).

What about the confidence level? By setting the confidence level we specify, how certain we want to be about our confidence interval. So far I used 95%. What happens if I calculate the confidence intervals for my original sample of 100 records, but with varying confidence level?

In this case the average is always the same, because only one sample was used.

As we can see, increasing the confidence level widens the interval. In other words, the more certain we want to be about the interval containing the real population average, the wider the interval has to be.

Sampling distribution

So far so good, but where does this magic confidence interval actually come from? It is calculated by the theoretical analysis of the sampling distribution (not to be confused with sample distribution):

Sample distribution is when we collect one sample of size k and calculate a certain feature for every element in the sample. It is a distribution of k values of the feature in one sample.
Sampling distribution is when we independently collect n samples, each of size k, and calculate the sample proportion for each sample. It is the distribution of n sample proportions.

Imagine I collect all samples of size 100 from my population and I calculate the sample proportion for each sample. This is the sampling distribution. Now I randomly choose one number from this sampling distribution. Note that this is equivalent to what I did before: choosing one random sample of size 100 and calculating its sample proportion.

According to Central Limit Theorem, sampling distribution is approximately normal with the mean equal to the population proportion. Here is the visualisation of the sampling distribution:

The black vertical line shows the mean of the sampling distribution. This is also the real population proportion. The grey area covers the middle 95% of the distribution mass (within 2 standard deviations from the mean).

When we choose one sample and calculate the sample proportion, there are two possibilities:

With 95% probability, we were lucky and the sample proportion is within the grey area. In that case, the real population proportion is not further than 2 standard deviations from our estimate.
With 5% probability, we were unlucky and the sample proportion is outside the grey area. In that case, the real population proportion is further than 2 standard deviations from our estimate.

So with the confidence of 95% we can say that the real population proportion is within 2 standard deviations from our sample proportion. We can see now that these 2 standard deviations of the sampling distribution define our confidence interval at the confidence level of 95%.

Smaller confidence level would make the grey area narrower, and the confidence interval would shrink as well. Larger confidence level makes the grey area, and the confidence interval, larger.

To look more closely at the sampling distribution, I generated sampling distributions for all combinations of “n samples of size k”, where n and k are the elements of the set {25, 50, 100, 200, 400, 800, 1600, 3200}. This is only an approximation, since the real sampling distributions would contain many more samples.

Here is the heatmap showing the mean of each sampling distribution (this should be approximately the same as the real population proportion):

We can see that there is some variability in the top left part of the heatmap, which corresponds to small sample sizes and small numbers of samples. The bottom right part of the heatmap shows much less variability. As we increase the sample size and number of samples, the mean of the sampling distribution approaches numbers around 0.933.

Here is the heatmap showing the standard deviation for each sampling distribution:

We can clearly see how the standard deviation decreases when we increase the sample size. This is consistent with the previous observation, that the confidence interval decreases when the sample size is increased.

Let’s also see the histograms of all the sampling distributions:

Here we can see the following patterns:

All histograms indeed seem to be centered around approximately the same number.
The more samples we include, the more normal the sampling distribution appears. This happens because with more samples the real sampling distribution is better approximated.
The larger the sample size, the narrower the sampling distribution (i.e. smaller standard deviation).

The estimation vs. the real value

Let’s go back to my original question. What is the proportion of reference strings in APA style, that are successfully matched to the original DOIs of the records they were generated from? So far we observed the following:

A small sample of 100 gave the estimate 0.92 (confidence interval 0.8668-0.9732)
A larger samples of 1000 gave the estimate 0.932 (confidence interval 0.9164-0.9476)
The means of sampling distributions seem to slowly approach 0.933

So what is the real population proportion in my case? It is 0.933005. As we can see, the estimations were fairly close, and the intervals indeed contain the real value.

Now I can also calculate the confidence interval for each sample in my sampling distributions, and then the fraction of the intervals that contain the real population proportion (I expect these numbers to be close to the confidence level 95%). Here is the heatmap:

We can see that for larger sample sizes indeed the fractions are high. The fraction is not always above 95%, as we would expect, especially for smaller sample sizes. One of the reasons is that when we calculate the confidence interval, we approximate the standard deviation of the population with the standard deviation of the sample. This is not always a reliable estimate, especially for small samples. This suggests that sample sizes of at least 1000-2000 should be used.

Be careful

Some important things to remember:

Aggregate functions. As mentioned before, apart from estimating the proportion, a similar procedure can be applied for estimating the average of a certain numeric feature.
(Lack of) certainty. Remember that the confidence level < 1. This means that we are never sure that our confidence interval contains the true population proportion. If for any reason you need to be 100% sure, just process the entire dataset.
Randomness, a.k.a. “stirring before tasting”. The sample has to be chosen randomly. Beware of assuming that the dataset is shuffled and taking the first 1000 rows!
Sample size. We know already that the larger the sample, the better. As a rule of thumb, using sample sizes < 30 makes the estimates, including the interval, rather unreliable.
Skewness. In general, the more skewed the original feature distribution, the larger sample we need. In case of the proportion, the sample should contain at least 5 data points of each value of the feature (passes/doesn’t pass the test).
Generalization. The sample average/proportion can be used as an estimate for the population average/proportion, but only the population it was drawn from. This means that if we applied any filters before sampling (which is equivalent to sampling from a subset passing the filter), we can reason only about the filtered subset of the data.
Reproducibility. This is more of an engineering concern. In short, all the analyses we do should be reproducible. In the context of sampling it means, at the very least, that we should record the samples we use.

URLs and DOIs: a complicated relationship

Joe Wass — Fri, 04 Nov 2016 00:00:00 +0000

As the linking hub for scholarly content, it’s our job to tame URLs and put in their place something better. Why? Most URLs suffer from link rot and can be created, deleted or changed at any time. And that’s a problem if you’re trying to cite them.

Thus the Crossref DOI was born: an Identifier which is Persistent, which means that it’s designed to live forever (or, as Geoff Bilder rather more prosaically puts it, as long as we do), and also Resolvable, which means that you can click on it. A DOI is a URL, but it’s imbued with special properties. I say special, not magical, because all of the things that make Crossref DOIs what they are, are obtained through agreements and common standards rather than any kind of magic.

As part of the development of Crossref Event Data I’ve been doing some research about the relationship between DOIs and URLs. It’s a problem we have to solve in order to make Event Data work, but it’s a much broader and more interesting story, and the results have wide applicability. I’ll be telling this story at PIDapalooza. If you’re interested in Persistent Identifiers you should go and registration is open, though hurry, as it’s next week and in Rejkjavik, Iceland!

This is also a story in progress. As I write not all of the data is in, and we can be certain that it will evolve in ways we have no idea about. It’s also quite long but I’ll do my best to disqualify it from the bedtime reading list.

Full circle

Crossref was established just over fifteen years ago with the purpose of forming the linking hub between publishers. Our job was — and still is — to register content for publishers and then continue to work with them to ensure their DOIs always point to the right location of the content. To do this we need to do one main thing: send people in the right direction when they click on a DOI, and know which direction to point them in.

Today, linking is still an important part of what Crossref does, but we do a huge amount more. One of the new things we’re working on is Crossref Event Data. It’s a service for tracking how and where people use scholarly content (such as articles) across the web and social media. Early research suggested that if we limited ourselves to just looking for DOIs we wouldn’t find much. Instead we broadened our aims a little: rather than looking for mentions of registered content exclusively via their DOIs, we look for them via the most suitable mechanism. In most cases this means the actual URL of the Item. So we have come full circle: we started linking DOIs to URLs. Now we’re trying to link URLs back to DOIs.

Which URL are we talking about here? The Crossref Guidelines say:

DOI-routed reference links enabled by Crossref must resolve to a response page containing no less than complete bibliographic information about the target content …

http://www.crossref.org.turing.library.northwestern.edu/02publishers/59pub_rules.html

This is what’s referred to as the Landing Page. Every Landing Page has a URL. Usually when you want to read information about an Article, it’s the Landing Page that you’re looking at. I should also say at this point that when I say Article I mean any item of Crossref Registered Content with a DOI. So the same applies to books, chapters, conference proceedings etc. But as most items are Articles, I’ll stick with that for now.

I’m going to make some assumptions. Unfortunately, and I don’t want to spoil the surprise here, they all turn out to be false. They’re all reasonable assumptions, though, and you would be forgiven for thinking, or at least wishing, that they were true.

So suspend your disbelief and follow me down the rabbit-hole…

Assumption 1: A DOI points directly to a Landing Page URL

When you click on a DOI you are taken to the Article Landing Page. It seems like a perfectly valid assumption to think that you are taken directly there.

The DOI system is essentially a big lookup table. In the first column is the DOI and in the second column is the URL. Publishers request that we register each item’s DOI and supply us with the URL it should point to. We work with CNRI and the International DOI Foundation to keep the system running and it means that when you, the reader at home, click on a DOI, you end up on the article’s Landing Page.

It would be very convenient if our assumption were true. If we wanted to turn a URL back into an article page, we could just swap the two columns and find the DOI by looking up the URL.

It turns out that it’s not quite so simple.

The Landing Page is under control of the publisher, as is the URL that they supply us with. They don’t need to supply us with the final landing page URL, only with one that leads to the landing page.

HTTP redirects

When you request a URL, either by typing it into your browser or by clicking on a link, your browser contacts the server and gets a reply. That reply can be “200 OK, here’s your page”, “303, look over there” or the dreaded “404, I can’t find it”. Other HTTP response codes are available, including well-known classics such as 201, 500 and 418.

If it’s a 303, your browser will follow the redirect URL. The response that comes back from that redirect could be another 303. You could end up following a whole chain of redirects. You wouldn’t notice anything, except having to wait an extra few milliseconds.

Extraordinary diversity

Crossref was created by a group of publishers who needed a way to link between articles. It was an ambitious goal: create a central system with which any publisher can integrate their own systems; one that allows linking to any article no matter who published it. Today we have over 5,000 members and counting, all contributing to our metadata engine. And up to 2 million DOIs are resolved every day, by all kinds of people and systems. Our wide range of members means a wide range of systems with a wide range of designs.

This brings an extraordinary diversity of behavior. If we want to make observations about DOIs we can’t just take a random sample of the over 80 million. Instead, we need to take a sample of DOIs per Publisher System. Even taking a sample per publisher might not do the job because some publishers run a variety of systems.

Experiment 1: Does Crossref know all Landing Pages?

By NASA / Paul Riedel (Great Images in NASA: Home - info - pic) [Public domain], via Wikimedia Commons

Hypothesis: Crossref knows the Landing Page URL for all DOIs.

For a sample of Items, we can follow the DOI link all the way through to the Landing Page, following any redirects, then compare the final Landing Page URL to the one that Crossref knows about. If there are extra redirects, that means that the one we have on file isn’t the final one.

We need to tighten up the terminology at this stage:

DOI URL - The full DOI, e.g. https://doi-org.turing.library.northwestern.edu/10.5555/12345678 .
Resource URL - The URL that Crossref has on file (stored in our system). This is where the browser is initially redirected.
Destination URL - The URL that we end up at if we follow all the redirects.
Article Landing Page - The page that represents the item. If everything works, this should be the same as the Destination URL.

The reason we’re talking about the Destination URL as distinct from the Article Landing Page when they should be the same thing will become clear later. Consider yourself foreshadowed.

So let’s re-word our hypothesis:

Hypothesis: The Destination URL is the same as the Resource URL.

Method: A sample of DOIs was taken (most items updated in 2016, all from 2009 or earlier). The Resource URL was obtained for all of them. The DOIs were split by the domain name of the Resource URL (to give a good coverage of all Publisher systems). A sample of Resource URLs was followed per domain, at least 200 (or fewer if that exceeds the number of DOIs available). Where there were HTTP redirects they were followed.

Observations:

Number of Items sampled Destination URL: 253,381
Number where Resource URL = Destination URL: 46,995 or 19.96%

Conclusion: Not all Resource URLs are the same as the Destination URL by a long shot. Crossref does not automatically know every landing page URL.

Now we know the truth about our first assumption: DOIs don’t point directly to Landing Pages. If we want to reverse Landing Pages back into DOIs, we’re going to need to go a bit deeper…

Interlude

But first, an interlude with some information about publishers, owners, and systems, because now seems like the right time to do it.

Assumption 2: You can tell the publisher of a DOI by looking at its prefix

This is a real one one that people believe. Again, it’s entirely understandable. People look at a DOI like https://doi-org.turing.library.northwestern.edu/10.1371/journal.pone.0136117.g001 , which takes them to PLoS and naturally assume that another DOI like https://doi-org.turing.library.northwestern.edu/10.1371/journal.pone.0136053.t003 — because it has the same prefix of 10.1371 — is also for a PLoS item.

Whilst this turns out to be true most of the time, it’s not true for all Items, which makes it a dangerous assumption to make.

It is true that every publisher is given a prefix. They can then register DOIs with this prefix. It is also true that Items can be transferred between publishers. Because DOIs are persistent, the prefix in the DOI doesn’t change. So you might find a DOI that belongs to a publisher that has an unexpected prefix. Publishers can also be bought and sold, merged and split, which means that whilst most publishers have a single prefix, some, like Elsevier, have several. Take the case of Elsevier, who has 26 at the time of writing (you can see this in Elsevier’s entry in the Crossref Metadata API).

Every Item has an ‘owner prefix’ in addition to the prefix in the DOI. The owner prefix is the same as the DOI prefix when the Item is created, but over time, as articles are transferred, that can change to indicate that it is owned by another publisher.

Every Item has a DOI, and every DOI has a prefix. But every Item also has an Owner Prefix (you can check this in the Metadata API in the ‘prefix’ field).

So Assumption 2 has been laid to rest. The only thing you can tell from looking at a DOI is that it is, in fact, a DOI (you can tell by the “10.” index code).

Why do we care about identifying publishers anyway?

A Fair Test

We fundamentally want to conduct a fair test. The reason we can’t just take a random sample from the set of all DOIs is that there are lots of members who all do things slightly differently. Therefore we need to take a sample per publisher ‘system’. The word ‘system’ is a bit fuzzy, but my assumption is that two articles in the same system will behave the same way so we can treat them the same.

We also know that each Crossref member may be running more than one system, or a mixture. Therefore just looking at the owner of a DOI may not give accurate results if we want to conduct a survey of all the systems out there.

There’s no perfect answer, but the approach I’m taking is to look at the domain name of the Resource URL. We often find lots of subdomains for the same publisher, for example, “psw.sagepub.com”, “pol.sagepub.com”, “psx.sagepub.com” and “bpi.sagepub.com”. It’s clear that these are all operated by Sage, but they might or might not all be running on different ‘systems’.

Therefore I’m splitting DOIs up into groups based on the domain of their Resource URL. It may turn out that some publishers use a single system running on many domains, or it may turn out that some publishers use a different system for each domain they use. The key point is to find a sampling technique that broadly works, and that allows us to explore and differentiate, as keenly as possible, the variety of systems and behaviours.

Why all the redirects?

Curious minds might at this stage be wondering about all these extra redirects. Surely it’s extra stuff for the publisher to maintain. Why don’t they just point the DOI directly to the landing page?

The answer must be prefaced by repeating that there is a huge number of publishers, running a variety of systems, so we’ll never be able to completely answer that. But some humble suggestions:

They might want to be able to change the URLs of the Landing Pages. It may be easier to update their internal systems than send the update to Crossref, especially in bulk.
Different parts of their technology stack may be owned by different parts of the company, or outsourced. It’s easier to define internal boundaries than to co-ordinate business units and cross an external one.
A publisher may run a mix of different technology. As part of their systems integration process, they set up a redirect server to make everything work together.
A publisher assigns DOIs to articles but also has their own internal IDs. They maintain their own DOI-to-internal-ID lookup service.

Internal DOI resolvers

That last point is an interesting one. The DOI system is the canonical “DOI-to-URL resolver”. That doesn’t prevent publishers from running their own. Indeed, many do.

To take a real example of PLoS, an Open Access publisher who registers lots of content with Crossref. To follow one of their DOIs we go on the following journey of redirects:

https://doi-org.turing.library.northwestern.edu/10.1371/journal.pone.0164910
http://dx.plos.org/10.1371/journal.pone.0164910
http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0164910
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0164910

Given that the last step uses a DOI, this suggests that they use the DOI as an internal identifier. All those redirects were for some purpose, but they weren’t mapping a DOI to an internal ID. This is therefore not an internal DOI resolver.

Another example from JAMA Surgery:

In this case we see a mapping from the DOI 10.1001/archsurg.142.7.595 to the ID 487551.

Can we define a heuristic for this pattern? Yes, but not a perfect one. My test is this:

Does the resource URL contain the DOI?
If so, does it redirect to a different destination URL?
If so, does the destination URL not contain the DOI?

The last step is important, because we can’t really say the publisher is running a DOI resolver if they use the DOI all the way through.

It’s not perfect and no doubt has false negatives. But we’re just trying to find out whether some publishers run their own DOI resolver systems.

Experiment 2: Determine how widespread use of internal DOI resolvers is:

Hypothesis: Some publishers run their own DOI resolvers.

Method: A number of Destination URLs were sampled per Resource URL Domain. If the Resource URL contains the DOI but the Destination URL doesn’t, that’s marked as a Publisher DOI resolver redirect.

Observations:

Number of Items sampled with Resource URL and Destination URL: 253,381
Number of Items that appear to be DOI resolvers: 166,352 = 65.6%

Conclusions: Some publishers run their own DOI resolvers.

This isn’t of much practical use, but it’s interesting to know, and hints at the way the Crossref system and DOIs are integrated with Publishers’ systems. Now that we’ve got a little insight into the reasons that publishers might run their own DOI resolvers, we can resume our journey of assumptions.

Assumption 3: We can find the Landing Page for Every DOI

Now we know that we can’t just use the lookup table in reverse, but have to follow the links all the way to their destination. Does this approach actually work?

This is a pretty big question and we need to be clear about what we mean by ‘every’ DOI. The set of DOIs I’m using (although I’m using a subset) is “all DOIs in our Metadata API that are found in doi.org”.

What is a DOI? Geoff Bilder went over it in the DOI-like-strings blog post earlier this year. The definition I’m working to here is:

A DOI is an identifier for an item of content registered in the DOI system.

That is, if you resolve the DOI on https://doi-org.turing.library.northwestern.edu/ and it’s recognised, that counts as a DOI. I’m working from the set of DOIs found in the Crossref system as I’m primarily concerned with Crossref DOIs. However, we collaborate closely with DataCite.

Back to our assumption: “we can find the Landing Page for every DOI”. The answer is that we can, most of the time. But because Crossref Event Data has to work as well as possible, and therefore work with as many DOIs as possible, we have to scour all the nooks and crannies.

Assumption 4: Every DOI points somewhere unique

Stop me when you find the deliberate mistake:

Every Item corresponds to a different thing
Every Item has a single DOI
Every DOI is different
Every DOI points to a landing page
Therefore every DOI points to a different landing page

Two things immediately suggest themselves:

“Every item has a single DOI” should be true, but it isn’t. We find that sometimes two DOIs are assigned to the same item. This can happen when publications change hands between publishers, or when mistakes are made, or for a variety of other reasons. We also find that in some cases Publishers registered a DOI for the metadata and one for the article abstract. The two DOIs point to the same place. In some cases where there were two DOIs registered for the same thing we create an Alias.

When we alias a DOI we simply say “this DOI should actually point to this one”. Both DOIs still exist, and both still point to the ‘correct’ thing, it’s just that they both point to the same place. If we have two DOIs pointing to the same place, then there isn’t a one-to-one mapping, and Assumption 4 is incorrect.

Experiment 4: Aliased DOIs

Hypothesis: There isn’t a one-to-one mapping between DOIs and URLs because some DOIs are aliased to others.

Method: We collected a sample of Resource URLs from the DOI API. We count how many DOIs are classified as Aliases in the DOI system.

Observations:

From a sample of 11,227,458 DOIs
14,566 are aliased to others, or 0.129%

Conclusion: There aren’t many aliases. But there are some, and we should be aware of them.

Experiment 5: Duplicate Resource URLs

Hypothesis: There isn’t a one-to-one mapping between DOIs and URLs because some DOIs have duplicate Resource URLs.

Method: A sample of Resource URLs was collected from the DOI API. We counted how many DOIs have Resource URLs that aren’t unique. We subtract the number of deleted DOIs because all deleted DOIs have the same resource URL.

Observations:

From a sample size of 11,227,458
a total of 112,195 have duplicate resource URLs, or 0.99%
of these duplicates, 77,896 have the ‘deleted’ URL
leaving 34,229, or 0.30% having non-unique Resource URLs

Conclusion: A small number of DOIs have duplicate Resource URLs, even if we exclude those that have been deleted, which means that not every DOI can have a unique URL.

Assumption 5: The Landing Page is the same as the Destination Page.

HTTP has a very neat system for doing redirects. If it were that simple, then we could easily look up every Destination page and confidently say that it was the Landing Page. Not so.

Cookies

Web browsers aren’t the only tools that use HTTP. Most programming languages have HTTP capabilities built in.

Using cookies is a requirement of some websites, but it’s not a requirement of HTTP. Most websites use cookies in some way or another. When you log into a site, you expect cookies. But when you’re just browsing there isn’t any technical need. A small number of websites absolutely require cookies to be enabled to use the site, even if you’re just browsing and not logged in. Unfortunately, this includes some publishers.

Requiring cookies to use a publisher site means that you can’t fully resolve a DOI without enabling cookies. Most tools out there don’t. Some privacy-conscious people quite reasonably don’t enable cookies from all sites.

Using cookies when resolving a DOI adds considerable overhead and isn’t fool-proof.

Let’s try a quick experiment to see when we land up on a cookie page. Here’s an example page that tells us that we should have enabled cookies: http://www-tandfonline-com.turing.library.northwestern.edu/action/cookieAbsent . It’s reachable from the DOI: https://doi-org.turing.library.northwestern.edu/10.1016/j.envhaz.2007.09.007 .

Experiment 6: Some DOIs can’t be resolved without cookies

Hypothesis: We can’t resolve some DOIs to the Landing Page using standard tools because cookies are required.

Method: A sample of DOIs was taken per Resource URL Domain. They were resolved by following HTTP links. Where the Destination URL contains the word ‘cookie’, we mark that as a DOI requiring a cookie.

Observations:

A sample of 253,381 DOIs were resolved following HTTP redirects where necessary
a total of 6305 resolved to a page with ‘cookie’ in the URL or 2.48%

Conclusion: There are cookies at play for at least 2.48% of DOIs. This is probably a very conservative estimate, as we’re using a blunt tool looking for ‘cookie’ in the URL.

Cookies Required

For one DOI I found, the publisher system set cookies, then sent us on a series of redirects which set cookies that expired in the past and then, as far as I can tell, checked whether or not they were sent back. My working hypothesis is that it was profiling the behaviour to see what browser I was using.

I have also seen javascript-based redirects. This is where a web page loads a javascript file, which executes and sends the browser onto another URL. This seems to be to be a browser detection method. There is no way you can follow these DOIs without actually using a real browser.

This is a problem for Crossref Event Data. We can’t fire up a browser and follow every DOI: it isn’t practical. When I tried this for a sample as an experiment I got an email from another publisher who was worried that we were scraping data (good bot operators always put contact details in their request headers!).

The Crossref member rules leave some wiggle-room about whether this is allowed, but for the Event Data service, we can say that it’s a physical impossibility to collect all Event Data for DOIs like this.

Bring in the Browser

To quantify the size of the problem, we need to bring in a web browser. If we assume that some Publishers design their sites to work only with real browsers, that’s what we’ll use. Luckily there are web browsers packaged up into an automatable package, and we can use these to visit the DOI.

Using one of these is considerably slower than just following link headers.

I have split the ‘destination’ concept into two:

Naïve destination URL: The URL that you get from following HTTP redirects acccording to the HTTP specification
Browser destination URL: The URL that you get from letting a browser follow the DOI doing whatever a browser does.

Rather than defining a complicated spectrum of types of DOI resolution behaviour, I am classifying DOIs into two groups: those where standard HTTP redirects are sufficient and everything else.

The method I am using is to resolve a sample of URLs using the browser. I can then compare the Naïve Destination URL with the Browser Destination URL. If they are the same, then I didn’t need to use the browser after all. If they give a different result however, I trust the Browser one better and declare that DOI to require a browser to resolve.

Again, I took a sample of DOIs per Resource URL domain.

Experiment 7: Quantify proportion of DOIs that require a browser to redirect

Hypothesis: A number of DOIs can’t be resolved with standard tools but instead require a browser.

Method: A sample of DOIs was selected per Resource URL domain. The links were followed using standard HTTP and using a browser. Where the URLs between the two were different, the DOI was counted as requiring a browser to resolve.

Observations:

A total of 59,453 items were followed both using the Naïve and Browser methods.
Of these 5,883 items have a different URL between the two methods, or 9.88%

Conclusion: We can’t rely on the Naïve redirect, and would have to fire up the browser in about 10% of cases in the sample.

Other gnarly things

There are one or two supplementary gnarly things that crop up.

First, session IDs are sometimes embedded in the URL. This is a tracking technique similar to cookies, but instead of sending cookies, which are invisible to the user, a unique code is placed on the end of the URL. This means that everyone gets a different URL. The most popular of these is the JSESSIONID, which is used by servers in the Java ecosystem. An example URL is:

http://onlinelibrary.wiley.com.turing.library.northwestern.edu/doi/10.1002/047084289X.rn00615.pub3/abstract;jsessionid=0D1B7AC4689A494E0EA78BD2F0A710C4.f04t04

We can easily remove these if they appear at the end of a URL. Sometimes they occur in the middle of a URL, as above. Sometimes they appear as query parameters:

http://jpharmsci.org/action/consumeSharedSessionAction?SERVER=WZ6myaEXBLGvmNGtLlDx7g%3D%3D&MAID=npYBLvZTaUI3JTHw%2BH63WQ%3D%3D&JSESSIONID=aaajjhdDL5ssK6d1HHrFv&ORIGIN=207988872&RD=RD

In this case we make no attempt to remove them. These URLs won’t be any use for matching, and we have to acknowledge that and move on.

Interpreting the results

All the above experiments involved taking as many DOIs as we had time for, gathering the Resource URLs, and then grouping the DOIs per Resource URL Domain. A sample of DOIs was investigated per each Resource URL domain to give the best chance at even coverage. The above figures have been presented as a proportion of the sampled data-set.

Now it’s time to draw some practical conclusions. I grouped the results per Resource URL Domain, so I can say that “for this domain, X% of DOIs was deleted, or aliased, or whatever”. This means that we can look at the statistics for a given domain and work out the best method for working with DOIs that belong to it.

I have created histograms of domains by their various proportions.

Our first chart is histogram of Resource URL Domains where the Naïve Destination = the Resource URL. Each domain is given a proportion which represents how many DOIs sampled on that domain have a Landing Page equal to the Resource URL.

There’s a clear bimodal distribution here. The conclusion here is “most domains require you to follow the link to find the destination URL“. Furthermore, the domains are consistent: there are virtually no domains that have a mix of DOIs that behave differently.

Our second chart is a histogram of Resource URLs where the Browser-based redirect = the Naive URL. Each domain is given a proportion which represents how many DOIs sampled on that domain require us to fire up a browser.

Overwhelmingly, the Browser Redirect URL is the same as the Naïve Redirect URL, meaning that we don’t need to fire up the browser, we can just use the Naïve URL, which is much easier to compute. There are some resource URL domains which require every DOI to be followed in a browser rather than just following links.

We know from this that we don’t have to use the browser most of the time. There is a small number of domains where we’re unsure (under 500) and a small number of domains where we know that we have to use a browser. This means we can focus our efforts.

There are lots of DOIs and they all behave differently.

There are thousands of publishers out there registering DOIs. There are thousands of domains. Some publishers have lots of domains. This makes it impossible to make many general observations about DOIs.

You can’t tell anything by looking at the DOI

Just by looking at the DOI you can’t tell who published it, or which publisher’s system is hosting it. Therefore you can’t tell how it’s going to behave.

We’ve looked at five kinds of URLs:

The DOI itself
The Resource URL
The “naïve” redirect URL
The “browser” redirect URL
The Article Landing Page

In some cases, the Resource URL, naïve redirect URL, browser redirect and Article Landing Page are the same. In some cases they aren’t. Of these, the fifth is somewhat mythical.

DOIs fall into classifications

Each DOI falls into a category, most preferable first:

The Resource URL is the same as the Landing Page.
The Landing Page can be discovered by following HTTP redirects.
The Landing Page can be discovered by firing up a web browser to follow redirects.
The Landing Page can’t be determined.

We can predictively group DOIs

We can group DOIs by their Resource URLs and take a sample per Resource URL Domain. If all samples for a domain behave a certain way, we can place the DOIs into one of the above four groups with a probability.

We’ll never know the full story.

Because of the diversity of Publisher Systems and the long history of Crossref DOIs, we’ll never be able to describe exactly what’s going on for all DOIs.

What next?

We’re continuing to develop Crossref Event Data. The part of the system that handles turning URLs back into DOIs will never be perfect, but we know from this research that we can at least work with a subset.

I’m also working on another project which will attempt to reverse a Landing Page URL back into a DOI by looking at the metadata on the Landing Page. You can read about it here. Ultimately we’re going to have to take a blended approach. Building a useful set of Landing Page URL to DOI mappings will be part of the mix.

As Event Data matures we’ll be sharing all the datasets automatically as part of our infrastructure, including our DOI-to-URL mapping.

And any members reading, please make your DOIs as easy to follow as possible! Please don’t require JavaScript or cookies when resolving DOIs.

If you’re read this far, perhaps you’re as interested in DOIs as we are. There’s a lot more to say on the subject, but that’s enough for now. See you at PIDapalooza!

Image Credits

All images from Wikipedia Commons. Click or hover on the image to see the attribution.

Using AWS S3 as a large key-value store for Chronograph

Joe Wass — Tue, 02 Aug 2016 00:00:00 +0000

One of the cool things about working in Crossref Labs is that interesting experiments come up from time to time. One experiment, entitled “what happens if you plot DOI referral domains on a chart?” turned into the Chronograph project. In case you missed it, Chronograph analyses our DOI resolution logs and shows how many times each DOI link was resolved per month, and also how many times a given domain referred traffic to DOI links per day.

We’ve released a new version of Chronograph. This post explains how it was put together. One for the programmers out there.

Big enough to be annoying

Chronograph sits on the boundary between normal-sized data and large-enough-to-be-annoying-size data. It doesn’t store data for all DOIs (it includes only those that are used on average once a day), but it has information on up to 1 million DOIs per month over about 5 years, and about 500 million data points in total.

Storing 500 million data points is within the capabilities of a well-configured database. In the first iteration of Chronograph a MySQL database was used. But that kind of data starts to get tricky to back up, move around and index.

Every month or two new data comes in for processing, and it needs to be uploaded and merged into the database. Indexes need to be updated. Disk space needs to be monitored. This can be tedious.

Key values

Because the data for a DOI is all retrieved at once, it can be stored together. So instead of a table that looks like

<td>
<span >2010-01-01</span>
</td>
<td>
<span >5</span>
</td>

<td>
<span >2010-02-01</span>
</td>
<td>
<span >7</span>
</td>

<td>
<span >2010-03-01</span>
</td>
<td>
<span >3</span>
</td>

10.5555/12345678

Instead we can store

<td>
{&#8220;2010-01-01&#8221;: 5, &#8220;2010-02-01&#8221;: 7, &#8220;2010-03-01&#8221;: 3}
</td>

10.5555/12345678

This is much lighter on the indexes and takes much less space to store. However, it means that adding new data is expensive. Every time there’s new data for a month, the structure must be parsed, merged with the new data, serialised and stored again millions of times over.

After trials with MySql, MongoDB and MapDB, this approach was taken with MySQL in the original Chronograph.

Keep it Simple Storage Service Stupid

In the original version of Chronograph the data was processed using Apache Spark. There are various solutions for storing this kind of data, including Cassandra, time-series databases and so on.

The flip side of being able to do interesting experiments is wanting them to stick around without having to bother a sysadmin. The data is important to us, but we’d rather not have to worry about running another server and database if possible.

Chronograph fits into the category of ‘interesting’ rather than ‘mission-critical’ projects, so we’d rather not have to maintain expensive infrastructure if possible.

I decided to look into using Amazon Web Services Simple Storage Service (AWS S3) to store the data. AWS itself is a key-value store, so it seems like a good fit. S3 is a great service because, as the name suggests, it’s a simple service for storing a large number of files. It’s cheap and its capabilities and cost scale well.

However, storing and updating up to 80 million very small keys (one per DOI) isn’t very clever, and certainly isn’t practical. I looked at DynamoDB, but we still face the overhead of making a large number of small updates.

Is it weird?

In these days of plentiful databases with cheap indexes (and by ‘these days’ I mean the 1970s onward) it seems somehow wrong to use plain old text files. However, the whole Hadoop “Big Data” movement was predicated on a return to batch processing files. Commoditisation of services like S3 and the shift to do more in the browser have precipitated a bit of a rethink. The movement to abandon LAMP stacks and use static site generators is picking up pace. The term ‘serverless architecture’ is hard to avoid if you read certain news sites.

Using Apache Spark (with its brilliant RDD concept) was useful for bootstrapping the data processing for Chronograph, but the new code has an entirely flat-file workflow. The simplicity of not having to unnecessarily maintain a Hadoop HDFS instance seems to be the right choice in this case.

Repurposing the Wheel

The solution was to use S3 as a big hash table to store the final data that’s served to users.

The processing pipeline uses flat files all the way through from input log files to projections to aggregations. At the penultimate stage of the pipeline blocks of CSV per DOI are produced that represent date-value pairs.

<td>
2010-01
</td>
<td>
2010-01-01,05<br /> 2010-02-01,02<br /> 2010-01-03,08<br /> &#8230;
</td>

<td>
2010-02
</td>
<td>
2010-02-1,10<br /> 2010-02-01,7<br /> 2010-02-03,22<br /> &#8230;
</td>

10.5555/12345678

At the last stage, these are combined into blocks of all dates for a DOI

<td>
2010-01
</td>
<td>
2010-01-01,05<br /> 2010-02-01,02<br /> 2010-01-03,08<br /> &#8230;<br /> 2010-02-1,10<br /> 2010-02-01,7<br /> 2010-02-03,22<br /> &#8230;
</td>

10.5555/12345678

The DOIs are then hashed into 12 bits and stored as chunks of CSV

day-doi.csv-chunks_8841:

10.1038/ng.3020
2014-06-24,4
2014-06-25,4
2014-06-26,3
...
10.1007/978-94-007-2869-1_7
2012-06-01,12
2012-06-02,8
...
10.1371/journal.pone.0145509
2016-02-01,13
2016-02-02,75
2016-02-03,30
...

There are 65,536 (0x000 to 0xFFFF) possible files, each with about a thousand DOIs worth of data in each.

When the browser requests data for a DOI, it is hashed and then the request for the appropriate file in S3 is made. The browser then has to perform a linear scan of the file to find the DOI it is looking for.

This is the simplest possible form of hash table: simple addressing with separate linear chaining. The hash function is a 16-bit mask of MD5, chosen because of availability in the browser. It does a great job of evenly distributing the DOIs over all 65,536 possible files.

Striking the balance

In any data structure implementation, there are balances to be struck. Traditionally these concern memory layout, the shape of the data, practicalities of disk access and CPU cost.

In this instance, the factors in play included the number of buckets that need to be uploaded and the cost of the browser downloading an over-large bucket. The size of the bucket doesn’t matter much for CPU (as far as the user is concerned it takes about the same time to scan 10 entries as it does 10,000), but it does make a difference asking user to download a 10kb bucket or a 10MB one.

I struck the balance at 4096 buckets, resulting in files of around 100k, which is the size of a medium sized image.

It works

The result is a simple system that allows people to look up data for millions of DOIs, without having to look after another server. It’s also portable to any other file storage service.

The approach isn’t groundbreaking, but it works.

HTTPS and Wikipedia

Joe Wass — Tue, 31 May 2016 00:00:00 +0000

This is a joint blog post with Dario Taraborelli, coming from WikiCite 2016.

In 2014 we were taking our first steps along the path that would lead us to Crossref Event Data. At this time I started looking into the DOI resolution logs to see if we could get any interesting information out of them. This project, which became Chronograph, showed which domains were driving traffic to Crossref DOIs.

You can read about the latest results from this analysis in the “Where do DOI Clicks Come From” blog post.

Having this data tells us, amongst other things:

where people are using DOIs in unexpected places
where people are using DOIs in unexpected ways
where we knew people were using DOIs but the links are more popular than we realised

By the time the ALM Workshop 2014 rolled around there was some preliminary data and we realised that Wikipedia came into the third category. There are lots of DOIs in Wikipedia and people click them!

I met with Dario Taraborelli, head of research at the Wikimedia Foundation, and shared the data. Dario — who co-authored in 2010 the Altmetrics Manifesto — has been interested in understanding how scholarly citations are used in Wikipedia. Over the years, Wikipedia contributors have made extensive use of references to the scientific literature using DOIs, and by doing so they have created a resource that represents today in many ways the “front matter to all research”. There is growing interest in the community in understanding how DOIs are being used in Wikipedia and in non traditional scholarship.

During our discussions the subject of Wikipedia’s gradual transition to HTTPS was raised: we anticipated that this change would affect our data gathering.

Changes

When you’re reading webpage and click on a link to another page, your web browser will usually tell the server of that second page the last page you were on. This forms the basis of trackers like Google Analytics.

In the days before HTTPS, the next site would know the full URL that you were previously on. With the change to HTTPS, this was reduced to just sending the domain name and not the full URL, or no data at all if you click from an HTTPS page to HTTP.

DOI hyperlinks are just like any other hyperlink, and are mostly HTTP not HTTPS.

Up until 2015, Wikipedia was served over HTTP, only switching to HTTPS when users were logged in or if they requested it. The Wikimedia Foundation started planning to move to HTTPS and we knew that if they did that, and continued to use HTTP DOIs then we would lose valuable research data.

A Plan

We decided that the best course of action was to try and change the DOIs in Wikipedia to use HTTPS. Simple, right?

After some further research, Dario posted a proposal on how to mitigate the impact of the HTTPS rollout, to make sure that Wikipedia can still signal its importance as a traffic source, while preserving the privacy of its users. Discussion followed and the conclusion was to change the format of every single DOI on Wikipedia, which fortunately could be done without having to edit millions of pages. You can read the full story in this post from a year ago.

The result of this effort was that well in advance of the HTTPS switchover, the DOI links were ready to continue reporting referral data.

The Switch

In June 2015 the Wikimedia foundation made the announcement that they were finalising the switch, and that within a few weeks all traffic would be HTTPS.

We held our breath. Would it work? Would we lose all referral data from Wikipedia sites? In February 2016 the last piece of the puzzle fell into place as Wikipedia gained a ‘meta referrer’ tag to explicitly specify how they would like referrers to be sent: a detailed report on the effect of this change is coming up on the Wikimedia Foundation’s blog.

The results

As detailed in the last blog post the traffic that we measured coming from Wikipedia doesn’t seem to have slowed down during 2015:

I’d call that a success! Over the period covered in the graph, Wikipedia remained prominent as a non-publisher referral of traffic to DOIs.

Looking at the balance of HTTP vs HTTPS traffic coming from wikipedia.org, the switchover was dramatic:

Thank you to Dario Taraborelli, Nemo (Federico Leva), Aaron Halfaker, Alex Stinson and everyone who put in this effort.

I’ll leave the last word to Dario:

It’s great to see this data. It shows that the switchover happened successfully, which better protects the privacy of our users whilst still reporting the fact that Wikipedia is a prominent source of traffic. This is important validation of the increasing role that Wikipedia plays in the education and scientific community.

Where do DOI clicks come from?

Joe Wass — Thu, 19 May 2016 00:00:00 +0000

As part of our Event Data work we’ve been investigating where DOI resolutions come from. A resolution could be someone clicking a DOI hyperlink, or a search engine spider gathering data or a publisher’s system performing its duties. Our server logs tell us every time a DOI was resolved and, if it was by someone using a web browser, which website they were on when they clicked the DOI. This is called a referral.

This information is interesting because it shows not only where DOI hyperlinks are found across the web, but also when they are actually followed. This data allows us a glimpse into scholarly citation beyond references in traditional literature.

Last year Crossref Labs announced Chronograph, an experimental system for browsing some of this data. We’re working toward a new version, but in the meantime I’d like to share the results for 2015 and some of 2016. We have filtered out domains that belong to Crossref member publishers to highlight citations beyond traditional publications.

Top 10 DOI referrals from websites in 2015

This chart shows the top 10 referring non-primary-publisher domains of DOIs per month. Note that if browsers don’t send the referrer (e.g. from an HTTPS page), we don’t get to find out. Because the top 10 can be different month to month, the total number of domains mentioned can be more than 10. Subdomains are combined, which means that, for example, the wikipedia.org entry covers all Wikipedia languages. This chart covers all of 2015 and the first two months of 2016.

The top 10 referring domains for the period:

webofknowledge.com
baidu.com
serialssolutions.com
scopus.com
exlibrisgroup.com
wikipedia.org
google.com
uni-trier.de
ebsco.com
google.co.uk

It’s not surprising to see some of these domains here: for example serialssolutions.com and exlibrisgroup.com are effectively proxies for link resolvers, Baidu and Google are incredibly popular search engines which would show up anywhere. But it is exciting to see Wikipedia ranked amongst these. For more detail look out for the new Chronograph.

HTTP vs HTTPS in 2015

We’ve also seen a steady increase in HTTPS referral traffic, i.e. people clicking on DOIs from sites that are using HTTPS. While it is still dwarfed by HTTP, there was a steady uptick throughout 2015.

This chart shows HTTP vs HTTPS referrals per day, which shows up the weekly spikes. It doesn’t include resolutions where we don’t know the referrer.

Increasing numbers of people are moving to HTTPS for reasons of security, privacy and protection from tampering. Google has announced plans to take HTTPS into account when ranking search results. Wikipedia has moved exclusively to HTTPS, and I’ll be telling the story of how Crossref and Wikipedia collaborated in an upcoming blog post.

Chronograph

Another version of Chronograph will be available soon. It will contain full data for all non-primary-publisher referring domains. Stay tuned!

Crossref & the Art of Cartography: an Open Map for Scholarly Communications

Jennifer Lin — Fri, 08 Jan 2016 00:00:00 +0000

In the 2015 Crossref Annual Meeting, I introduced a metaphor for the work that we do at Crossref. I re-present it here for broader discussion as this narrative continues to play a guiding role in the development of products and services this year.

Metadata enable connections

At Crossref, we make research outputs easy to find, cite, link, and assess through DOIs. Publishers register their publications and deposit metadata through a variety of channels (XML, CSV, PDF, manual entry), which we process and transform into Crossref XML for inclusion into our corpus. This data infrastructure which makes possible scholarly communications without restrictions on publisher, subject area, geography, etc. is far more than a reference list, index or directory.

If research builds on what came before, one could claim that the process of knowledge production is partly the story of the very relationships between results disseminated (i.e., publications). So let’s consider each publication as a node in a graph where each has a coordinate and is connected by its citations to other publications (as well those that cite it). Additionally, each is associated with a set of people and places, along with a whole host of elements involved in the research and dissemination process.

But take a wider berth, and we begin to capture relationships between all such contributing agents and objects involved in the research process. Here we find an array of entities belonging to the scholarly graph, including different types of research artifacts, publisher and journal, funders, ORCIDs, peer reviews, publication status updates (corrections, retractions, etc.), citations, license information, additional URLs (machine destinations, hosting platforms, etc.), underlying data, software and protocols, materials, discussions and blog posts, recommendations, reference work mentions, etc. The entities on the graph multiply at an even higher rate as researchers share more outputs across more channels. And over time, the graph expands exponentially, producing a webbing that is far more dense and far more vast than we can currently imagine. Perhaps even to the point we realize Borges’ story where a cartographer builds a map so large it replicates the territory itself (On Exactitude in Science)!

From graph to cartography

At the heart of Borges’s poignant story is the map. Crossref’s graph of scholarly communications could be seen in the same light. It has a representational aspect, which is not purely abstract and can be visualized. Here, a map becomes an incredibly potent metaphor. Each link enabled by publisher-deposited metadata is a new street, bridge, or highway that takes us to a particular place (i.e., entity) of interest. These roads lead to articles, researchers, funders, institutions, etc., and in doing so, make them discoverable. They tell a story about the roles of each in the broader research in the landscape dotted with a plethora of places.

The scholarly web has a growing corpus of more than 78 million publications at this very moment registered with Crossref. On average ten to fifteen thousand new objects appear every day. Maps are all the more essential for getting around in a bewildering environment of new and unfamiliar places, even for known ones in areas of exploding growth. They are critical for orienteering, discovering relationships, identifying sets of associated objects, naming new neighborhoods that emerge (i.e., new research specialties), etc. And if each connection on the map is seen as an event, maps can also represent micro-narratives about the research process and the agents involved. A multi-dimensional map containing all these entities, which serves as an evolving representation of spacetime that is constantly updated and always available, would finally begin to depict the process of scholarly activity as a dynamic, evolving, almost living system.

An open map for scholarly communication

Crossref builds such a scholarly map of the research enterprise and makes it openly available for the entire research ecosystem. Call this a meta map or, more recently, call it metastructure. No matter what name it goes by we call it infrastructure at Crossref.

Crossref’s open map for scholarly communications is a core part of the open information infrastructure for scholarly research. Crossref map data are open, portable, as well as licensed and provisioned for maximum reuse to serve the whole community. This open resource has two entrances: one for humans, another for machines. The Crossref REST API enables machines to traverse this environment and mine it in equal measure to the humans behind them. It is configured so that a robot can learn, a phone can access, and platforms can be built.

OpenStreetMap and Google Maps, both widely used and mature infrastructure maps, are instructive examples when we consider a map of this kind for scholarly communications. Map data can be represented in unlimited ways, depending on any variety of needs and users. Third parties can add content via interactive layers that tell different stories such as health expenditure by country based on GDP and coral reefs at risk. They have a broad base of users across business models from philanthropic services aimed at disaster relief (Refugeemaps.eu) to commercial entities providing drivers with locations on open parking spaces (AppyParking on Google Map, PocketParker on OpenStreetMap). They power platforms and services that build maps for others (MapQuest, MapBox). They have applications far beyond the business of maps. For example, Place picker is a Google Maps widget that supports easy auto-complete the entry of any place or location on a mobile app where typing is a chore. And as far use cases close to home, the two have served as raw data for academic research (ex: workflow for generating multi-agent traffic simulation scenarios, automatic classification of GPS trajectories for transportation modes, etc.).

In kind, the Crossref infrastructure map also supports: the development of any variety of new maps which re-present the data, the makers of map platforms that power the research enterprise, tools that use map data, as well as academic research (bibliometrics). We extract slices of data of common interest from the map and add them as additional layers by which anyone can access and create applications on or across these bands of data:

Contributors (authors, editors, reviewers)
Funding information (funding body, grant number)
Trial & study information (clinical trials registry number, registered report, replication study)
Publication history (versions, updates, revisions, corrections, retractions, dates received/accepted/published)
Peer review (status, type, reviews)
Access indicators (publication license for text & data mining, machine mining URLs)
Resources & associated research artifacts (preprints, figures & tables, datasets, software, protocols, research resource IDs)
Activity surrounding the publication (peer reviews, comments & discussions, bookmarks, social shares, recommendations).

Today, the map powers a host of public and commercial organisations alike for a wide range of scholarly and non-scholarly purposes:

<td style="border: 1px solid #ffffff;">
<ul>
<li>
<span >Indexing services</span>
</li>
<li>
<span >Publishing vendors</span>
</li>
<li>
<span >Peer review systems</span>
</li>
<li>
<span >Reference manager systems</span>
</li>
<li>
<span >Lab & diagnostics suppliers</span>
</li>
<li>
<span >Info management systems</span>
</li>
<li>
<span >Educational tools</span>
</li>
<li>
<span >Data analytics systems</span>
</li>
<li>
<span >Literature discovery services</span>
</li>
</ul>
</td>

Publishers
Funders
Research institutions
Archives & repositories
Research councils
Data centres
Professional networks
Patent offices
Registration Agencies

We will follow up this post to highlight a cross-section of these consumers in the Crossref map ecosystem and elaborate on what & how they have built from our data. An infrastructure map offers endless potential to third parties across publishers, funders, research institutions, and vendors working to serve the scholarly research enterprise.

The art of cartography

In the Crossref Product Management team, we have ambitious plans for map enhancements this year. They focus on expanding information density and ease of access to the data. In the former case, we will introduce a new class of locations where activity surrounding the publications are occurring when we launch the DOI Event Tracker. We will also initiate an extensive publisher campaign to achieve full metadata deposit completeness across our membership. No one can keep pace with the sheer volume of research activity happening online nor wander the Lonely Web of research alone. The more metadata publishers provide for a publication, the more roads lead to its map location. After all, discoverability is closely associated with connectedness on a map. And finally, in the latter case, we will refresh and enhance the user interface to make it more powerful for humans to traverse the ever-changing landscape (as easily as the REST API enables machines!).

I gratefully acknowledge the feedback received from the following who served as generous and insightful sounding boards: Virginia Barbour, Theo Bloom, Martin Eve, Daniel S. Katz, Amye Kenall, Catriona MacCullum, Cameron Neylon, Mark Patterson, Kristen Ratan, Carly Strasser, and Kaitlin Thaney.

Crossref Labs plays with the Raspberry Pi Zero

Joe Wass — Wed, 02 Dec 2015 00:00:00 +0000

If you’re anything like us at Crossref Labs (and we know some of you are) you would have been very excited about the launch of the Raspberry Pi Zero a couple of days ago. In case you missed it, this is a new edition of the tiny low-priced Raspberry Pi computer. Very tiny and very low-priced. At $5 we just had to have one, and ordered one before we knew exactly what we want to do with it. You would have done the same. Bad luck if it was out of stock.

We love the way DOIs are being used in Wikipedia, but you probably already know that by now. Not only is it a brilliant source of information, mostly well cited, it’s also an organic living thing, with countless people and bots working together on countless articles. Our live stream of edits that cite (or uncite) DOIs shows new scholarly literature unfold, as it happens. From new articles to new references to improved citations to edit wars to bots cleaning up all the mess, it captivates everyone we show it to. The latest version has a live chart to show exactly how much activity is going on.

Crossref works in five ways: Rally, Tag, Run, Play, and Make and this definitely comes under ‘Play’. By the time our Raspberry Pi Zero arrived it was clear what we had to do. We ordered a servo, a driver board and a wireless adapter and got to work.

We have some new neighbours in the basement. Oxford Hackspace is a community of people who want to work on projects from electronics to metalwork, hack things to improve them or find out how they work. A diverse bunch who at the last visit were working on squeezing unprecedented color capabilities from the 30 year old ZX Spectrum, a nixie tube display, a smartphone controlled doorbell and a robotic glockenspiel. They let us use their soldering iron to solder a few header pins.

A bit of hacky Python, a pictureframe and lots of duck tape later, we have a live display of how many DOIs are cited and uncited per hour. It updates live every minute, fetches the latest numbers from the Wikipedia DOI citation stream and moves the hand.

(For the worried engineers amongst you, rest assured that sufficient duck tape was added after this picture)

It’s extraordinary to think that a fully fledged computer with very capable specifications can be manufactured and sold for $5. Within the space of a lunchtime we had it up and running, all connected and fetching data over the internet via wireless. A generation ago you would have had to use punched cards, send them by post and load them in by hand. The live stream would have been at least a month behind.

It now sits in our Oxford office reminding us that DOIs Aren’t Just for Traditional Bibliographies. Below Geoff Bilder’s reminder about what happens when you have too many standards (they’re telephone plugs from round the world).

You can find source code and instructions on the github repository so you can make your own if you want.

DOIs in Reddit

Joe Wass — Wed, 30 Sep 2015 00:00:00 +0000

Skimming the headlines on Hacker News yesterday morning, I noticed something exciting. A dump of all the submissions to Reddit since 2006. “How many of those are DOIs?”, I thought. Reddit is a very broad community, but has some very interesting parts, including some great science communication. How much are DOIs used in Reddit?

(There has since been a discussion about this blog post on Hacker News)

We have a whole strategy for DOI Event Tracking, but nothing beats a quick hack or is more irresistible than a data dump.

What is a DOI?

If you know what a DOI is, skip this! The DOI system (Digital Object Identifier) is a link redirection service. When a publisher puts some content online they could just hand out the URL. But the URL can change, and within a very short space of time, link-rot happens. DOIs are designed to fight link rot. When a publisher mints a DOI to an article they just published, they can change the article’s URL and then update the DOI to point to the new place. DOIs are persistent. They are URLs. They’re also identifiers (kind of like ISBNs), and they’re used in scholarly publishing as to do citations.

Crossref is the DOI registration agency for scholarly publishing. That means mostly things like journal articles. There are other registration agencies, for example, DataCite, who do DOIs for research datasets. But at this point in time, most DOIs are Crossref’s.

What does finding DOIs in Reddit mean?

It means someone used a DOI to cite something! DOIs can be used for any kind of content, but because of the sheer volume of scientific publishing, lots of DOIs are for science. Having a DOI doesn’t say anything about quality or content. But it does indicate that the person who created the DOI probably intended it to be cited. We care because it means that every time a DOI is used a tiny bit of link-rot doesn’t have the opportunity to take hold. Every time something is discussed on Reddit and the DOI is used, it means that archaeologists using the data dump in 100 years will have identifiers to find the things being discussed, even if the web and URLs have long since crumbled to dust.

Or, more likely, in five year’s time when a few URLs will have shuffled around.

The results

DOIs have been used on Reddit since 2008 (the logs start in 2006). After a rocky start, we see hundreds being used per year.

That’s dozens per month.

The best subreddit to find DOIs is /r/Scholar, followed by /r/science. And then a lot of others with one or two per year.

Opportunities

It’s great to see DOIs being used in Reddit. But let’s be honest, it’s not a massive amount.

We have a list of domains that our DOIs point to. They mostly belong to publishers, so every time we see a link to a domain on the list, there’s a chance (not a certainty) that the link could have been made using a DOI. We found a large number of these, orders of magnitude more than DOIs. We’re still crunching the data.

The data

The data is quite large. It’s a 40 Gigabyte download compressed, which comes to about 170 GB that uncompressed. It contains the submissions to reddit between 2006 and 2015, not the comments, so each data point represents a thread of conversation about a DOI.

Reproducibility (updated)

You can find the source code and reproduce the figures at http://github.com/crossref/reddit-dump-experiment. We use Apache Spark for this kind of thing.

The data and methodology are very experimental. You can download all results here:

https://s3-eu-west-1.amazonaws.com/crossref-labs-data/2015-10-06/reddit-dump-experiment.zip

It includes all data for charts in this post, as well as the full list of DOIs, the full list of URLs that could possibly have DOIs, and the full JSON input line for each of these.

More info

Read about our DOI Event Tracking strategy, including our live stream of Wikipedia citations.

Real-time Stream of DOIs being cited in Wikipedia

Joe Wass — Tue, 03 Mar 2015 00:00:00 +0000

TL;DR

Watch a real-time stream of DOIs being cited (and “un-cited!” ) in Wikipedia articles across the world: https://live-eventdata-crossref-org.turing.library.northwestern.edu/live.html

Background

For years we’ve known that the Wikipedia was a major referrer of Crossref DOIs and about a year ago we confirmed that, in fact, the Wikipedia is the 8th largest refer of Crossref DOIs. We know that people follow the DOIs, too. This despite a fraction of Wikipedia citations to the scholarly literature even using DOIs. So back in August we decided to create a Wikimedia Ambassador programme. The goal of the programme was to promote the use of persistent identifiers in citation and attribution in Wikipedia articles. We would do this through outreach and through the development of better citation-related tools.

Remember when we originally wrote about our experiments with the PLOS ALM code and how that has transitioned into the DOI Event Tracking Pilot? In those posts we mentioned that one of the hurdles in gathering information about DOI events is the actual process of polling third party APIs for activity related to millions of DOIs. Most parties simply wouldn’t be willing handle the load of a 100K API calls an hour. Besides, polling is a tremendously inefficient process, only a fraction of DOIs are ever going to generate events, but we’d have to poll for each of them, repeatedly, forever, to get an accurate picture of DOI activity. We needed a better way. We needed to see if we could reverse this process and convince some parties to instead “push” us information whenever they saw DOI related events (e.g. citations, downloads, shares, etc). If only we could convince somebody to try this…

Wikipedia DOI Events

In December 2014 we took the opportunity of the 2014 PLOS/Crossref ALM Workshop in San Francisco too meet with Max Klein and Anthony Di Franco where we kicked off a very exciting project.

There’s always someone editing a Wikipedia somewhere in the world. In fact, you can see a dizzying live stream of edits. We thought that given that there are so many DOIs in Wikipedia, that live stream may contain some diamonds (DOIs are made of diamond, that’s how they can be persistent). Max and Anthony went away and came back with a demo that contains a surprising amount of DOI activity.

That demo is evolving into a concrete service, called Cocytus. It is running at Wikimedia Labs monitoring live edits as you read this.

For now we’re feeding that data into the DOI Events Collection app (which is an off-shoot of the Chronograph project). We are in the process of modifying the Lagotto code so that we can instead push those events into the DOI Event Tracking Instance.

The first DOI event we noticed was delightfully prosaic: The DOI for “The polymath project” is cited by the Wikipedia page for “Polymath Project”. Prosaic perhaps, but the authors of that paper probably want to know. Maybe they can help edit the page.

Or how about this. Someone wrote a a paper about why people edit Wikipedia and then it was cited by Wikipedia. And then the citation was removed. The plot thickens…

We’re interested in seeing how DOIs are used outside of the formal scholarly literature. What does that mean? We don’t fully know, that’s the point. We have retractions in scholarly literature (and our Crossmark metadata and service allow publishers to record that), but it’s a bit different on Wikipedia. Edit wars are fought over … well you can see for yourself.

Citations can slip in and out of articles. We saw the DOI 10.1001/archpediatrics.2011.832 deleted from “Bipolar disorder in children”. If we’d not been monitoring the live feed (we had considered analysing snapshots of the Wikipedia in bulk) we might never have seen that. This is part of what non-traditional citations means, and it wasn’t obvious until we’d seen it.

You can see this activity on the Chronograph’s stream. Or check your favourite DOI. Please be aware that we’re only collecting newly added citations as of today. We do intend to go back and back-fill, but that may take some time- as it * cough * requires polling again.

Some Technical Things

A few interesting things that happened as a result of all this:

Secure URLs

SSL and HTTPS were invented so you could do things like banking on the web without fear of interception or tampering. As the web becomes a more important part of life, many sites are upgrading from HTTP to HTTPS, the secure version. This is not only because your confidential details may be tampered with, but because certain governments might not like you reading certain materials.

Because of this, some time ago, Wikipedia decided to embark on an upgrade to HTTPS last year, and they are a certain way along the path. The IDF, who are responsible for running the DOI system, upgraded to HTTPS this Summer, although most DOIs are referred to by HTTP still.

We met with Dario Taraborelli at the ALM workshop and discussed the DOI referral data that is fed into the Chronograph. We put two and two together and realised that Wikipedia was linking to DOIs (which are mostly HTTP) from pages which might be served over HTTPS. New policies in HTML5 specify that referrer URL headers shouldn’t be sent from HTTPS to HTTP (in case there was something secret in them). The upshot of this is, if someone’s browsing Wikipedia via HTTPS and click on a normal DOI, we won’t know that the user came from Wikipedia. Not a huge problem today, but as Wikipedia switches over to entirely secure, we’re going to miss out on very useful information.

Fortunately, the HTML5 specification includes a way to fix this (without leaking sensitive information). We discussed this with Dario, and he did some research, and came up with a suggestion, which got discussed. It’s fascinating to watch a democratic process like this take place and take part in it.

We’re waiting to see how the discussion turns out, and hope that it all works out so we can continue to report on how amazing Wikipedia is at sending people to scholarly literature.

How shall I cite thee?

Another discussion grew out of that process, and we started talking to a Wikipedian called Nemo (note to Latin scholars: we weren’t just talking to ourselves). Nemo (real name Federico Leva) had a few suggestions of his own. Another way to solve the referrer problem is by using HTTPS URLs (HTML5 allows browsers to send the referrer domain when going from HTTPS to HTTPS).

This means going back to all the articles that use DOIs and change them from HTTP to HTTPS. Not as simple as it sounds, and it doesn’t sound simple. We started looking into how DOIs were cited on Wikipedia.

After some research we found that there are more ways that we expected to cite DOIs.

First, there’s the URL. You can see it in action in this article. URLs can take various forms.

Second there’s the official template tag, seen in action here:

<ref name="SCI-20140731">{{cite journal |title=Sustained miniaturization and anatomical innovation in the dinosaurian ancestors of birds |url=http://www.sciencemag.org.turing.library.northwestern.edu/content/345/6196/562 |date=1 August 2014 |journal=[[Science (journal)|Science]] |volume=345 |issue=6196 |pages=562–566 |doi=10.1126/science.1252243 |accessdate=2 August 2014 |last1=Lee |first1=Michael S. Y. |first2=Andrea|last2=Cau |first3=Darren|last3=Naish|first4=Gareth J.|last4=Dyke}}</ref>

There’s a DOI in there somewhere. This is the best way to cite DOIs, firstly as it’s actually a proper traditional citation and there’s nothing magic about DOIs, secondly because it’s a template tag and can be re-rendered to look slightly different if needed.

Third there’s the old official DOI template tag that’s now discouraged:

<ref name="Example2006">{{Cite doi|10.1146/annurev.earth.33.092203.122621}}</ref>

And then there’s another one.

{{doi|10.5555/123456789}}

Knowing all this helps us find DOIs. But if we want to convert DOIs links in Wikipedia to use HTTPS, it means that there are more template tags to modify and more pages to re-render.

Nemo also put DOIs on the Interwiki Map which should make automatically changing some of the URLs a lot easier.

We’re very grateful to Nemo for his suggestions and work on this. We’ll report back!

The elephant in the room

Those of you who know how DOIs work will have spotted an unsecured elephant in the room. When you visit a DOI, you visit the URL, which hits the DOI resolver proxy server, which returns a message to your browser to redirect to the landing page on the publisher’s site.

Securely talking to the DOI resolver by using HTTPS instead of HTTP means that no-one can eavesdrop and see which DOI you are visiting, or tamper with the result and send you off to a different page. But the page you are sent to will be, in nearly all cases, still HTTP. Upgrading infrastructure isn’t trivial, and, with over 4000 members (mostly publishers), most Crossref DOIs will still redirect to standard HTTP pages for the foreseeable future.

You can keep as secure as possible by using HTTPS Everywhere.

Fin

There’s lots going on, watch this space to see developments. Thanks for reading this, and all the links. We’d love to know what you think.

Bootnote

Not long after this blog post was published we saw something very interesting.

That’s no DOI. We like interesting things, but they can panic us. This turned out to be a great example of why this kind of thing can be useful. A minute’s digging and we found the article edit:

It turns out that this was a typo: someone put a title when they should have put in a DOI. And, as the event shows, this was removed from the Wikipedia article.

Crossref’s DOI Event Tracker Pilot

Geoffrey Bilder — Mon, 02 Mar 2015 00:00:00 +0000

TL;DR

Crossref’s “DOI Event Tracker Pilot”- 11 million+ DOIs & 64 million+ events. You can play with it at: http://goo.gl/OxImJa

Tracking DOI Events

So have you been wondering what we’ve been doing since we posted about the experiments we were conducting using PLOS’s open source ALM code? A lot, it turns out. About a week after our post, we were contacted by a group of our members from OASPA who expressed an interest in working with the system. Apparently they were all about to conduct similar experiments using the ALM code, and they thought that it might be more efficient and interesting if they did so together using our installation. Yippee. Publishers working together. That’s what we’re all about.

So we convened the interested parties and had a meeting to discuss what problems they were trying to solve and how Crossref might be able to help them. That early meeting came to a consensus on a number of issues:

The group was interested in exploring the role Crossref could play in providing an open, common infrastructure to track activities around DOIs, they were not interested in having Crossref play a role in the value-add services of reporting on an interpreting the meaning of said activities.
The working group needed representatives from multiple stakeholders in the industry. Not just open access publishers from OASPA, but from subscription based publishers, funders, researchers and third party service providers as well.
That it was desirable to conduct a pilot to see if the proposed approach was both technically feasible and financially sustainable.

And so after that meeting, the “experiment” graduated to becoming a “pilot.” This Crossref pilot is based on the premise that the infrastructure involved in tracking common information about “DOI events” can be usefully separated from the value-added services of analysing and presenting these events in the form of qualitative indicators. There are many forms of events and interactions which may be of interest. Service providers will wish to analyse, aggregate and present those in a range of different ways depending on the customer and their problem. The capture of the underlying events can be kept separate from those services.

In order to ensure that the Crossref pilot is not mistaken for some sub rosa attempt to establish new metrics for evaluating scholarly output, we also decided eschew any moniker that includes the word “metrics” or synonyms. So the “ALM Experiment” is dead. Long live the “”DOI Event Tracker” (DET) pilot. Similarly PLOS’s open source “ALM software” has been resurrected under the name “Lagotto.”

The Technical Issues

Crossref members are interested in knowing about “events” relating to the DOIs that identify their content. But our members face a now-classic problem. There are a large number of sources for scholarly publications (3k+ Crossref members) and that list is still growing. Similarly, there are an unbounded number of potential sources for usage information. For example:

Supplemental and grey literature (e.g. data, software, working papers)
Orthogonal professional literature (e.g. patents, legal documents, governmental/NGO/IGO reports, consultation reports, professional trade literature).
Scholarly tools (e.g. citation management systems, text and data mining applications).
Secondary outlets for scholarly literature (institutional and disciplinary repositories, A&I services).
Mainstream media (e.g. BBC, New York Times).
Social media (e.g. Wikipedia, Twitter, Facebook, Blogs, Yo).

Finally, there is a broad and growing audience of stakeholders who are interested in seeing how the literature is being used. The audience includes publishers themselves as well as funders, researchers, institutions, policy makers and citizens.

Publishers (or other stakeholders) could conceivably each choose to run their own system to collect this information and redistribute it to interested parties. Or they can work with a vendor to do the same. But either case, they would face the following problems:

The N sources will change. New ones will emerge. Old ones will vanish.
The N audiences will change. New ones will emerge. Old ones will vanish.
Each publisher/vendor will need to deal with N source’s different APIs, rate limits, T&Cs, data licenses, etc. This is a logistical headache for both the publishers/vendors and for the sources.
Each audience will need to deal with N publisher/vendor APIs, rate limits, T&Cs, data licenses, etc. This is a logistical headache for both the audiences and for the publishers.
If publishers/vendors use different systems which in turn look at different sources, it will be difficult to compare or audit results across publishers/vendors.
If a journal moves from one publisher to another, then how are the metrics for that journal’s articles going to follow the journal?

And then there is the simple issue of scale. Most parties will be interested in comparing the data that they collect for their own content, with data about their competitors. Hence, if they all run their own system, they will each be querying much more than their own data. If, for example, just the commercial third-party providers were interested in collecting data covering the formal scholarly literature, they would each find themselves querying the same sources for the same 80 million DOIs. To put this into perspective, to refresh the data for 10 million DOIs once a month, would require sources to support ~ 14K API calls an hour. 60 million DOIs would require 100K API calls an hour. Current standard API caps for many of the sources that people are interested in querying hover around 2K per hour. We may see these sources lift that cap for exceptional cases, but they are unlikely to do so for many different clients all of whom are querying essentially the same thing.

These issues typify the “multiple bilateral relationships” problem that Crossref was founded to try and ameliorate. When we have many organisations trying to access the exact same APIs to process the exact same data (albeit to different ends), then it seems likely that Crossref could help make the process more efficient.

Piloting A Proposed Solution

The Crossref DET pilot aims to show the feasibility of providing a hub for the collection, storage and propagation of DOI events from multiple sources to multiple audiences.

Data Collection

Pull: DET will collect DOI event data from sources that are of common interest to the membership, but which are unlikely to make special efforts to accommodate the scholarly communications industry. Examples of this class of source include large, broadly popular services like FaceBook, Twitter, VK, Sina Weibo, etc.
Push: DET will allow sources to send DOI event data directly to Crossref in one of three ways:
- Standard Linkback: Using standards that are widely used on the web. This will automatically enable linkback-aware systems like WordPress, Moveable Type, etc. to alert DET to DOI events.
- Scholarly Linkback: A to-be-defined augmented linkback-style API which will be optimized to work with scholarly resources and which will allow for more sophisticated payloads including other identifiers (e.g. ORCIDs, FundRefs), metadata, provenance information and authorization information. This system could be used by tools designed for scholarly communications. So, for example, it could be used by publisher platforms to distribute events related to downloads or comments within their discussion forums. It could also be used by third party scholarly apps like Zotero, Mendeley, Papers, Authorea, IRUS-UK, etc. in order to alert interested parties in events related to specific DOIs.
- Redirect: DET will also be able to serve as a service discovery layer that will allow sources to push DOI event data directly to an appropriate publisher-controlled endpoint using the above scholarly linkback mechanism. This can be used by sources like repositories in order to send sensitive usage data directly to the relevant publishers.

Data Propagation

Parties may want to use the DET in order to propagate information about DOI events. The system will support two broad data propagation patterns:

one-to-many: DOI events that are commonly harvested (pulled) by the DET system from a single source will be distributed freely to anybody who queries the DET API. Similarly, sources that push DOI events via the standard or scholarly linkback mechanisms, will also propagate their DOI events openly to anybody who queries the DET API. DOI events that are propagated in either of these cases will be kept and logged by the DET system along with appropriate provenance information. This will be the most common, default propagation model for the DET system.
one-to-one: Sources of DOI events can also report (push) DOI event data directly to owner of the relevant DOI if the DOI owner provides & registers a suitable end-point with the DET system. In these cases, data sources seeking to report information relating to a DOI, will be redirected (with a suitable 30X HTTP status and relevant headers) to the end-point specified by the DOI owner. The DET system will not keep the request or provenance information. One-to-one propagation model is designed to handle use cases where the source of the DOI event has put restrictions on the data and will only share the DOI events with the owner (registrant) of the DOI. This use case may be used, for example, by aggregators or A&I services that want to report confidential data directly back to a publisher. The advantage of the redirect mechanism is that Crossref is not put into the position of having to secure sensitive data as said data will never reside on Crossref systems.

Note that the two patterns can be combined. So, for example, a publisher might want to have public social media events reported to the DET and propagated accordingly, but to also to private third parties report confidential information directly to the publisher.

So Where Are We?

So to start with, the DET Working Group has grown substantially since the early days and we have representatives from a wide variety of stakeholders. The group includes:

Cameron Neylon, PLOS
Chris Shillum, Elsevier
Dom Mitchell, Co-action Publishing
Euan Adie, Altmetric
Jennifer Lin, PLOS
Juan Pablo Alperin, PKP
Kevin Dolby, Wellcome Trust
Liz Ferguson, Wiley
Maciej Rymarz, Mendeley
Mark Patterson, eLife
Martin Fenner, PLOS
Mike Thelwell, U Wolverhampton
Rachel Craven, BMC
Richard O’Beirne, OUP
Ruth Ivimey-Cook, eLife
Victoria Rao, Elsevier

As well as the usual contingent of Crossref cat-herders including: Geoffrey Bilder, Rachael Lammey & Joe Wass.

When we announced the then-DET experiment, we said that one of the biggest challenges would be to create something that scaled to industry levels. At launch, we only loaded in about 317,500+ Crossref DOIs representing publications from 2014 and we could see the system was going to struggle. Since then Martin Fenner and Jennifer Lin at PLOS have been focusing on making sure that the Lagotto code scales appropriately and now it is currently humming along with just over 11.5 million DOIs for which we’ve gathered over 64 million “events.” We aren’t worried about scalability on that front any more.

We’ve also shown that third parties should be able to access the API to provide value added reporting and metrics. As a demonstration of this, PLOS configured a copy of its reporting software “Parascope” to point at the Crossref DET instance. The next step we’re taking is to start testing the “push” API mechanism and the “point-to-point redirect” API mechanism. For the push API, we should have a really exciting demo available to show within the next few days. And on the point-to-point redirect, we have a sub-group exploring how the point-to-point redirect mechanism could potentially be used for reporting COUNTER stats as a compliment to the Sushi initiative.

The other major outstanding task we have before us is to calculate what the costs will be of running the DET system as a production service. In this case we expect to have some pretty accurate data to go on as we will have had close to half a year of running the pilot with a non-trivial number of DOIs and sources. Note that the work group is concerned to ensure that the underlying data from the system remains open to all. Keeping this raw data open as seen as critical to establishing trust in the metrics and reporting systems that third parties build on the data. The group has also committed to leaving the creation of value-add services to third parties. As such we have been focusing on exploring business models based around service-level-agreement backed versions of the API to complement the free version of the same API. The free API will come with no guarantees of uptime, performance characteristics or support. For those users that depend on the API in order to deliver their services, we will offer paid-for SLA-backed versions of the free APIs. We can then configure our systems so that we can independently scale these SLA-backed APIs in order to meet SLA agreements.

Our goal is to have these calculations complete in time for the working group to make a recommendation to the Crossref board meeting in July 2015.

Until then, we’ll use CrossTech as a venue for notifying people when we’ve hit new milestones or added new capabilities to the DET Pilot system.

Introducing the Crossref Labs DOI Chronograph

Joe Wass — Mon, 12 Jan 2015 00:00:00 +0000

tl;dr http://chronograph.labs.crossref.org.turing.library.northwestern.edu

At Crossref we mint DOIs for publications and send them out into the world, but we like to hear how they’re getting on out there. Obviously, DOIs are used heavily within the formal scholarly literature and for citations, but they’re increasingly being used outside of formal publications in places we didn’t expect. With our DOI Event Tracking / ALM pilot project we’re collecting information about how DOIs are mentioned on the open web to try and build a picture about new methods of citation.

As part of the preparation for collaborating with Wikipedia, we looked at our statistics about when DOIs are clicked and discovered that Wikipedia was, over a two year period from 2012, the eighth largest referrer of DOIs. This means that not only does Wikipedia have a lot of DOIs, but people click them too. This bit of one-off data analysis (which surprised us) gave us enough of a prod to kickstart our collaboration with Wikipedia.

At the ALM Workshop 2014 in San Francisco we talked to some Wikipedians and bibliometricians and realised that we were sitting on a really interesting data-set and that it would be churlish not to share it. At the hackathon (read the report here) we started work on a service to gather information about DOIs and, a month later, we’re ready to unveil the DOI Chronograph.

Show me the goods

You can see:

Daily referrals (clicks) from top level domains, e.g. Wikipedia.org: http://chronograph.labs.crossref.org.turing.library.northwestern.edu/domain.html?domain=wikipedia.org

Daily referrals from specific subdomains, e.g. fr.wikipedia.org: http://chronograph.labs.crossref.org.turing.library.northwestern.edu/domain.html?domain=fr.wikipedia.org

Daily resolutions per DOI: http://chronograph.labs.crossref.org.turing.library.northwestern.edu/doi.html?doi=10.1787%2F20752288

And, the chart that kicked this all off: DOI referring domains league tables. This shows that Wikipedia is the 3rd or 4th non-traditional referrer of DOIs (i.e. excluding referrals from Publishers’ domains): http://chronograph.labs.crossref.org.turing.library.northwestern.edu/top.html

Try it out

Visit the Chronograph and give it a try chronograph.labs.crossref.org on your favourite DOI (everyone has one).

More data

Talking to a bibliometrician we also realised we can correlate other data for DOIs. We’re getting the issue date (approximately the publication date) from our own metadata, as well as the date that the Crossref metadata was updated. This gives interesting results, like the resolutions for 10.1038/ncomms2953, which peak after publication and then tails off. We are attempting to collect the following information:

daily resolution counts
day on which resolution was first successful
day on which it’s possible to resolve the DOI (we’ve got a bot running for new publications)
day on which the publisher says the article was published
day on which the metadata was most recently deposited with us
day on which the metadata was first deposited with us

We’re not there yet, but we’ve made a start and we’ve already got some pretty interesting data!

Weasel words

It’s a labs project so the usual weasel words apply. Specifically, we currently have the logs for 2012 to 2014 (we’re working at digging out the rest), and the referral information for 50 million DOIs (out of 71 million). That number will be higher by the time you read this. If your page is slow to load, be patient, as it’s currently working hard crunching numbers.

This project is focused on exploring the use of DOIs outside of the formal literature. As such, we are only looking at referrals from domains that do not appear to belong to primary publishers (i.e. our members). If you try a domain and it doesn’t work, it could be that the domain belongs to one of our members. If you’ve notice any mistakes, please email us at labs@crossref.org .

Finally, these numbers contain all DOI resolutions. That’s human clicks but also content negotiation to retrieve metadata, robots etc. We might try to filter them in future, but for now be aware that not every visitor is a human.

I’ll detail some of the the technical stuff (it’s very interesting) and what happened next with Wikipedia in a future post. Watch this space.

Citation needed

Geoffrey Bilder — Thu, 07 Aug 2014 00:00:00 +0000

Remember when I said that the Wikipedia was the 8th largest referrer of DOI links to published research? This despite only a fraction of eligible references in the free encyclopaedia using DOIs.

We aim to fix that. Crossref and Wikimedia are launching a new initiative to better integrate scholarly literature in the world’s largest public knowledge space, Wikipedia.

This work will help promote standard links to scholarly references within Wikipedia, which persist over time by ensuring consistent use of DOIs and other citation identifiers in Wikipedia references. Crossref will support the development and maintenance of Wikipedia’s citation tools on Wikipedia. This work will include bug fixes and performance improvements for existing tools, extending the tools to enable Wikipedia contributors to more easily look up and insert DOIs, and providing a “linkback” mechanism that alerts relevant parties when a persistent identifier is used in a Wikipedia reference.

In addition, Crossref is creating the role of Wikimedia Ambassador (modeled after Wikimedian-in-Residence) to act as liaison with the Wikimedia community, promote use of scholarly references on Wikipedia, and educate about DOIs and other scholarly identifiers (ORCIDs, PubMed IDs, DataCite DOIs, etc) across Wikimedia projects.

Starting today, Crossref will be working with Daniel Mietchen to coordinate Crossref’s Wikimedia-related activities. Daniel’s team will be composed of Max Klein and Matt Senate, who will work to enhance Wikimedia citation tools, and will share the role of Wikipedia ambassador with Dorothy Howard.

Since the beginnings of Wikipedia, Daniel Mietchen has worked to integrate scholarly content into Wikimedia projects. He is part of an impressive community of active Wikipedians and developers who have worked extensively on linking Wikipedia articles to the formal literature and other scholarly resources. We’ve been talking to him about this project for nearly a year, and are happy to finally get it off the ground.

-G

]7 Matt, Max and Daniel at #wikimania2014. Photo by Dorothy.

wikimania2014

♫ Researchers just wanna have funds ♫

Geoffrey Bilder — Thu, 10 Apr 2014 00:00:00 +0000

photo credit

Summary

You can use a new Crossref API to query all sorts of interesting things about who funded the research behind the content Crossref members publish.

Background

Back in May 2013 we launched Crossref’s FundRef service. It can be summarized like this:

Crossref keeps and manages a canonical list of Funder Names (ephemeral) and associated identifiers (persistent).
We encourage our members (or anybody, really- the list is available under A CC-Zero license waiver) to use this list for collecting information on who funded the research behind the content that our members publish.
We then ask that our members deposit this data in their normal Crossref metadata deposits.

And that was cool.

But then people started asking us awkward questions. Questions like “what can I do with the funder data?” and “how do I query it?”.

Stoopit people.

Can’t you just let us bask for a few minutes in the sunny glow of actually conceiving of and launching a project within a year?

But seriously, funders, were interested to see how they could use the funder metadata being collected in Crossref. In particular, some funding agencies were interested in being able to measure Key Performance Indicators (“KPIs” to management wonks) related to recent mandates such as the February 22nd 2013 OSTP memo, Public Access to the Results of Federally Funded Research. Two groups also approached us, CHORUS and SHARE. Both are interested in exploring how to build reporting tools for funders, institutions and researchers and each brought us a gigantic hairball of use-cases they were hoping we would be able to meet.

Conveniently, we were in the process of creating a revised, modern Crossref API that is entirely buzzword-compliant, and so we set to work…

We thought people might be interested in seeing what you can do with the Crossref REST API in relation to funding information and the expectations that are increasingly being attached to them. CHORUS is already using the Crossref REST API heavily and we expect that SHARE will soon start making use of it as well. The feedback from both groups has been very useful, but we are looking for broader feedback as well. The API is still in development, so now is your chance to help us shape it.

Brief Examples

Please note, the following are APIs calls, although you can copy and paste the URIs into your browser, the data is returned in a machine readable representation called JSON. If you want the results to look a little more presentable, we advise you install the JSONView plugin:

Firefox Users: JSONView
Chrome Users: JSONView

Also note that publishers have only just started to deposit the metadata needed for these APIs to work, so the data is currently sparse. We know that many of our members are working feverishly to populate more of the needed metadata, but this requires updates to the their manuscript tracking systems, production systems and hosting systems. It takes time.

But for now you can paste the relevant URIs below into your browser and see the results that we do have. Expect these numbers to increase sharply over the next few months

To start with, you might want to know how many articles in Crossref have FundRef metadata:

https://api-crossref-org.turing.library.northwestern.edu/v1/works?filter=has-funder:true&rows=0

You could then be interested in knowing how many works in Crossref use FundRef to credit the United States’ National Science Foundation (NSF) for funding their research? First you need to find out what the FundRef identifier is for the NSF:

https://api-crossref-org.turing.library.northwestern.edu/v1/funders?query=NSF

You can see that there are several entries that match “NSF”, and that the one we are looking for has the identifier https://doi-org.turing.library.northwestern.edu/10.13039/100000001. Remember, funding agency names can change frequently, the ID provides a persistent link to the funder even if their name changes.

If you are curious, you can see the details for the NSF entry, including its location, parent and child organisations:

https://api-crossref-org.turing.library.northwestern.edu/v1/funders/10.13039/100000001

Notice that the results also lists the work-count. This is the number of works in the Crossref metadata that list the US NSF as having funded the research.

So perhaps you would like to see the list of works. The following will list the first twenty:

https://api-crossref-org.turing.library.northwestern.edu/v1/funders/10.13039/100000001/works

You can page through the results with the offset argument:

https://api-crossref-org.turing.library.northwestern.edu/v1/funders/10.13039/100000001/works?offset=20
https://api-crossref-org.turing.library.northwestern.edu/v1/funders/10.13039/100000001/works?offset=40
...

How many works that have listed the NSF as a funder have license information:

https://api-crossref-org.turing.library.northwestern.edu/v1/funders/10.13039/100000001/works?filter=has-license:true&rows=0

Lets see the first batch that have license information:

https://api-crossref-org.turing.library.northwestern.edu/v1/funders/10.13039/100000001/works?filter=has-license:true

Lets look at the metadata for one of the DOIs returned:

https://api-crossref-org.turing.library.northwestern.edu/v1/works/10.1063/1.3593378

Interesting, the metadata shows an article published by AIP. It includes license information (CC-BY 3.0) as well as a link to the full text. If you follow the link to the full text, you can retrieve it:

http://link.aip.org.turing.library.northwestern.edu/link/applab/v98/i21/p216101/pdf/CHORUS

Wow- A pretty short article. But you can see that it does credit the NSF and that the award number recorded in the text is the same as the award number recorded in the FundRef section of the Crossref metadata. Yay.

You can see in the brief examples above that there is a lot of other metadata you may want to query on and explore. It can include ORCIDS, information about archiving arrangements- even abstracts. It all depends on what the Crossref member has decided to provide.

You can get a simple overview of what a Crossref member has provided by looking at a member summary. Here is an example for Hindawi:

https://api-crossref-org.turing.library.northwestern.edu/v1/members?query=hindawi

Note again that names are fickle, so the above query can also be accomplished using the member identifier like this:

https://api-crossref-org.turing.library.northwestern.edu/v1/members/98

Groovy init?

If you want more pointers on where you can learn how to use the API, read on…

The Future of the Crossref REST API.

Our aim is for the Crossref REST API to go into production this Summer (2014). As with most of our newer APIs, there will be a free API for public use and a paid for API for professional use. The only difference between the two will be that the professional version will come with a service level agreement (SLA) covering uptime, response time and support. Naturally, this also means that the professional one will be on dedicated hosting equipment so that we can meet these SLAs, whereas the performance of the free version will be subject to the vicissitudes inherent in using a shared, constrained resource (i.e. the server and network it is running on).

Again, the basics of the API are in place. It should be fairly stable, but we do reserve the right to make changes to it over the next few months. Please send us feedback.

— The Weasel

Many Metrics. Such Data. Wow.

Geoffrey Bilder — Mon, 24 Feb 2014 00:00:00 +0000

[

Crossref Labs loves to be the last to jump on an internet trend, so what better than than to combine the Doge meme with altmetrics?

Note: The API calls below have been superceeded with the development of the Event Data project. See the latest API documentation for equivalent functionality

Want to know how many times a Crossref DOI is cited by the Wikipedia?

http://det.labs.crossref.org.turing.library.northwestern.edu/works/doi/10.1371/journal.pone.0086859

Or how many times one has been mentioned in Europe PubMed Central?

http://det.labs.crossref.org.turing.library.northwestern.edu/works/doi/10.1016/j.neuropsychologia.2013.10.021

Or DataCite?

http://det.labs.crossref.org.turing.library.northwestern.edu/works/doi/10.1111/jeb.12289

Background

Back in 2011 PLOS released its awesome ALM system as open source software (OSS). At Crossref Labs, we thought it might be interesting to see what would happen if we ran our own instance of the system and loaded it up with a few Crossref DOIs. So we did. And the code fell over. Oops. Somehow it didn’t like dealing with 10 million DOIs. Funny that.

But the beauty of OSS is that we were able to work with PLOS to scale the code to handle our volume of data. Crossref contracted with Cottage Labs and we both worked with PLOS to make changes to the system. These eventually got fed back into the main ALM source on Github. Now everybody benefits from our work. Yay for OSS.

So if you want to know technical details, skip to Details for Propellerheads. But if you want to know why we did this, and what we plan to do with it, read on.

Why?

There are (cough) some problems in our industry that we can best solve with shared infrastructure. When publishers first put scholarly content online, they used to make bilateral reference linking agreements. These agreements allowed them to link citations using each other’s proprietary reference linking APIs. But this system didn’t scale. It was too time-consuming to negotiate all the agreements needed to link to other publishers. And linking through many proprietary citation APIs was too complex and too fragile. So the industry founded Crossref to create a common, cross-publisher citation linking API. Crossref has since obviated the need for bilateral linking arrangements.

So-called altmetrics look like they might have similar characteristics. You have ~4000 Crossref member publishers and N sources (e.g. Twitter, Mendeley, Facebook, CiteULike, etc.) where people use (e.g. discuss, bookmark, annotate, etc.) scholarly publications. Publishers could conceivably each choose to run their own system to collect this information. But if they did, they would face the following problems:

The N sources will be volatile. New ones will emerge. Old ones will vanish.
Each publisher will need to deal with each source’s different APIs, rate limits, T&Cs, data licenses, etc. This is a logistical headache for both the publishers and for the sources.
If publishers use different systems which in turn look at different sources, it will be difficult to compare results across publishers.
If a journal moves from one publisher to another, then how are the metrics for that journal’s articles going to follow the journal? This isn’t a complete list, but it shows that there might be some virtue in publishers sharing an infrastructure for collecting this data. But what about commercial providers? Couldn’t they provide these ALM services? Of course - and some of them currently do. But normally they look on the actual collection of this data as a means to an end. The real value they provide is in the analysis, reporting and tools that they build on top of the data. Crossref has no interest in building front-ends to this data. If there is a role for us to play here, it is simply in the collection and distribution of the data.

No, really, WHY?

Aren’t these altmetrics an ill-conceived and meretricious idea? By providing this kind of information, isn’t Crossref just encouraging feckless, neoliberal university administrators to hasten academia’s slide into a Stakhanovite dystopia? Can’t these systems be gamed?

FOR THE LOVE OF FSM, WHY IS CROSSREF DABBLING IN SOMETHING OF SUCH QUESTIONABLE VALUE?

takes deep breath. wipes spittle from beard

These are all serious concerns. Goodhart’s Law and all that… If a university’s appointments and promotion committee is largely swayed by Impact Factor, it won’t improve a thing if they substitute or supplement Impact Factor with altmetrics. Amy Brand has repeatedly pointed out, the best institutions simply don’t use metrics this way at all (PowerPoint presentation). They know better.

But yes, it is still likely that some powerful people will come to lazy conclusions based on altmetrics. And following that, other lazy, unscrupulous and opportunistic people will attempt to game said metrics. We may even see an industry emerge to exploit this mess and provide the scholarly equivalent of SEO. Feh. Now I’m depressed and I need a drink.

So again, why is Crossref doing this? Though we have our doubts about how effective altmetrics will be in evaluating the quality of content, we do believe that they are a useful tool for understanding how scholarly content is used and interpreted. The most eloquent arguments against altmetrics for measuring quality, inadvertently make the case for altmetrics as a tool for monitoring attention.

Critics of altmetrics point out that much of the attention that research receives outside of formal scholarly communications channels can be ascribed to:

Puffery. Researchers and/or university/publisher “PR wonks” over-promoting research results.
Innocent misinterpretation. A lay audience simply doesn’t understand the research results.
Deliberate misinterpretation. Ideologues misrepresent research results to support their agendas.
Salaciousness. The research appears to be about sex, drugs, crime, video games or other popular bogeymen.
Neurobollocks. A category unto itself these days.

In short, scholarly research might be misinterpreted. Shock horror. Ban all metrics. Whew. That won’t happen again.

Scholarly research has always been discussed outside of formal scholarly venues. Both by scholars themselves and by interested laity. Sometimes these discussions advance the scientific cause. Sometimes they undermine it. The University of Utah didn’t depend on widespread Internet access or social networks to promote yet-to-be peer-reviewed claims about cold fusion. That was just old-fashioned analogue puffery. And the Internet played no role in the Laetrile or DMSO crazes of the 1980s. You see, there were once these things called “newspapers.” And another thing called “television.” And a sophisticated meatspace-based social network called a “town square.”

But there are critical differences between then and now. As citizens get more access to the scholarly literature, it is far more likely that research is going to be discussed outside of formal scholarly venues. Now we can build tools to help researchers track these discussions. Now researchers can, if they need to, engage in the conversations as well. One would think that conscientious researchers would see it as their responsibility to remain engaged, to know how their research is being used. And especially to know when it is being misused.

That isn’t to say that we expect researchers will welcome this task. We are no Pollyannas. Researchers are already famously overstretched. They barely have time to keep up with the formally published literature. It seems cruel to expect them to keep up with the firehose of the Internet as well.

Which gets us back to the value of altmetrics tools. Our hope is that, as altmetrics tools evolve, they will provide publishers and researchers with an efficient mechanism for monitoring the use of their content in non-traditional venues. Just in the way that citations were used before they were distorted into proxies for credit and kudos.

We don’t think altmetrics are there yet. Partly because some parties are still tantalized by the prospect of usurping one metric for another. But mostly because the entire field is still nascent. People don’t yet know how the information can be combined and used effectively. So we still make naive assumptions such as “link=like” and “more=better.” Surely it will eventually occur to somebody that, instead, there may be a connection between repeated headline-grabbing research and academic fraud. A neuroscientist might be interested in a tool that alerts them if the MRI scans in their research paper are being misinterpreted on the web to promote neurobollocks. An immunologist may want to know if their research is being misused by the anti-vaccination movement. Perhaps the real value in gathering this data will be seen when somebody builds tools to help researchers DETECT puffery, social-citation cabals, and misinterpretation of research results?

But Crossref won’t be building those tools. What we might be able to do is help others overcome another hurdle that blocks the development of more sophisticated tools; getting hold of the needed data in the first place. This is why we are dabbling in altmetrics.

Wikipedia is already the 8th largest referrer of Crossref DOIs. Note that this doesn’t just mean that the Wikipedia cites lots of Crossref DOIs, it means that people actually click on and follow those DOIs to the scholarly literature. As scholarly communication transcends traditional outlets and as the audience for scholarly research broadens, we think that it will be more important for publishers and researcher to be aware of how their research is being discussed and used. They may even need to engage more with non-scholarly audiences. In order to do this, they need to be aware of the conversations. Crossref is providing this experimental data source in the hope that we can spur the development of more sophisticated tools for detecting and analyzing these conversations. Thankfully, this is an inexpensive experiment to conduct - largely thanks to the decision on the part of PLOS to open source its ALM code.

What Now?

Crossref’s instance of PLOS’s ALM code is an experiment. We mentioned that we had encountered scalability problems and that we had resolved some of them. But there are still big scalability issues to address. For example, assuming a response time of 1 second, if we wanted to poll the English-language version of the Wikipedia to see what had cited each of the 65 million DOIs held in Crossref, the process would take years to complete. But this is how the system is designed to work at the moment. It polls various source APIs to see if a particular DOI is “mentioned”. Parallelizing the queries might reduce the amount of time it takes to poll the Wikipedia, but it doesn’t reduce the work. Another obvious way in which we could improve the scalability of the system is to add a push mechanism to supplement the pull mechanism. Instead of going out and polling the Wikipedia 65 million times, we could establish a “scholarly linkback” mechanism that would allow third parties to alert us when DOIs and other scholarly identifiers are referenced (e.g. cited, bookmarked, shared). If the Wikipedia used this, then even in an extreme case scenario (i.e. everything in Wikipedia cites at least one Crossref DOI), this would mean that we would only need to process ~ 4 million trackbacks.

The other significant advantage of adding a push API is that it would take the burden off of Crossref to know what sources we want to poll. At the moment, if a new source comes online, we’d need to know about it and build a custom plugin to poll their data. This needlessly disadvantages new tools and services as it means that their data will not be gathered until they are big enough for us to pay attention to. If the service in question addresses a niche of the scholarly ecosystem, they may never become big enough. But if we allow sources to push data to us using a common infrastructure, then new sources do not need to wait for us to take notice before they can participate in the system.

Supporting (potentially) many new sources will raise another technical issue- tracking and maintaining the provenance of the data that we gather. The current ALM system does a pretty good job of keeping data, but if we ever want third parties to be able to rely on the system, we probably need to extend the provenance information so that the data is cheaply and easily auditable.

Perhaps the most important thing we want to learn from running this experimental ALM instance is: what it would take to run the system as a production service? What technical resources would it require? How could they be supported? And from this we hope to gain enough information to decide whether the service is worth running and, if so, by whom. Crossref is just one of several organisations that could run such a service, but it is not clear if it would be the best one. We hope that as we work with PLOS, our members and the rest of the scholarly community, we’ll get a better idea of how such a service should be governed and sustained.

Details for Propellerheads

Warning, Caveats and Weasel Words

The Crossref ALM instance is a Crossref Labs project. It is running on R&D equipment in a non-production environment administered by an orangutang on a diet of Redbulls and vodka.

So what is working?

The system has been initially loaded with 317,500+ Crossref DOIs representing publications from 2014. We will load more DOIs in reverse chronological order until we get bored or until the system falls over again.

We have activated the following sources:

PubMed

DataCite

PubMedCentral Europe Citations and Usage

We have data from the following sources but will need some work to achieve stability:

Facebook

Wikipedia

CiteULike

Twitter

Some of them are faster than others. Some are more temperamental than others. WordPress, for example, seems to go into a sulk and shut itself off after approximately 1,300 API calls.

In any case, we will be monitoring and tweaking the sources as we gather data. We will also add new sources as we get requested API keys. We will probably even create one or two new sources ourselves. Watch this blog and we’ll update you as we add/tweak sources.

Dammit, shut up already and tell me how to query stuff.

You can login to the Crossref ALM instance simply using a Mozilla Persona (yes, we’d eventually like to support ORCID too). Once logged-in, your account page will list an API key. Using the API key, you can do things like:

http://det.labs.crossref.org.turing.library.northwestern.edu/api/v5/articles?ids=10.1038/nature12990

And you will see that (as of this writing), said Nature article has been cited by the Wikipedia article here:

https://en.wikipedia.org/wiki/HE0107-5240#cite_ref-Keller2014_4-0;

PLOS has provided lovely detailed instructions for using the API- So, please, play with the API and see what you make of it. On our side we will be looking at how we can improve performance and expand coverage. We don’t promise much- the logistics here are formidable. As we said above, once you start working with millions of documents, the polling process starts to hit API walls quickly. But that is all part of the experiment. We appreciate your helping us and would like your feedback. We can be contacted at:

Easily add publications to your ORCID profile

Geoffrey Bilder — Thu, 24 Jan 2013 00:00:00 +0000

You can now easily search for publications and add them to your ORCID profile in the new beta of Crossref Metadata Search (CRMDS). The user interface is pretty self-explanatory, but if you want to read about it before trying it, here is a summary of how it works.

When you go to to CRMDS, you will see that there is now a small ORCID sign-in button on the top right-hand side of the screen.