Bibliometrics on Crossref

Crossref metadata for bibliometrics

Ginny Hendricks — Fri, 21 Feb 2020 00:00:00 +0000

Our paper, Crossref: the sustainable source of community-owned scholarly metadata, was recently published in Quantitative Science Studies (MIT Press). The paper describes the scholarly metadata collected and made available by Crossref, as well as its importance in the scholarly research ecosystem.

Containing over 106 million records and expanding at an average rate of 11% a year, Crossref’s metadata has become one of the major sources of scholarly data for publishers, authors, librarians, funders, and researchers. The metadata set consists of 13 record types, including not only traditional types, such as journals and conference papers, but also data sets, reports, preprints, peer reviews, and grants. The metadata is not limited to basic publication metadata, but can also include abstracts and links to full text, funding and license information, citation links, and the information about corrections, updates, retractions, etc. This scale and breadth make Crossref a valuable source for research in scientometrics, including measuring the growth and impact of science and understanding new trends in scholarly communications. The metadata is available through a number of APIs, including REST API and OAI-PMH.

In the paper, we describe the kind of metadata that Crossref provides and how it is collected and curated. We also look at Crossref’s role in the research ecosystem and trends in metadata curation over the years, including the evolution of its citation data provision. We summarize the research that used Crossref’s metadata and describe plans that will improve metadata quality and retrieval in the future.

Bridging Identifiers at PIDapalooza

Joe Wass — Mon, 22 Jan 2018 00:00:00 +0000

Hello from sunny Girona! I’m heading to PIDapalooza, the Persistent Identifier festival, as it returns for its second year. It’s all about to kick off.

One of the themes this year is “bridging worlds”: how to bring together different communities and the identifiers they use. Something I really enjoyed about PIDapalooza last year was the variety of people who came. We heard about some “traditional” identifier systems (at least, it seems that way to us): DOIs for publications, DOIs for datasets, ORCIDs for researchers. But, gathered in Reykjavik, under dark Icelandic skies, I met oceanographic surveyors assigning DOIs to drilling equipment, heard stories of identifiers in Chinese milk production and consoled librarians trying navigate the identifier landscape.

In addition to the usual scholarly publishing and science communication crowd, it was encouraging to see a real diversity of people from different walks of life encounter the same problems and work on them them collaboratively. The thing that brought everyone together was the understanding that if we’re going to reliably reference things – be they researchers, articles they write, or ships they sail – we need to give them identifiers. And those identifiers should be as good as possible: persistent, resolvable, interoperable.

Who cares about PIDs?

At the turn of the century, a handful of publishers came together to create Crossref (or CrossRef as it was in those days). It was becoming increasingly important to be able to store references in machine-readable format, but publishers were faced with a problem. If an author wants to cite an article, they’ll do so without worrying who published it. This means they needed an identifier system that worked across all publishers. Thus the Crossref DOI was born.

Today we’re heading toward 10,000 members, and the thing that they have in common is that they all produce scholarly content and care about how it’s referenced. As a trade association, we effectively act on behalf of all of our members, allowing them to register their content, share metadata and links, and assign an identifier.

But there’s a whole world out there. Publications have never been the be-all and end-all of scholarship, but they have been a backbone. But more and more scholarship, especially science, is done outside journal publishing. Sometimes it’s done on platforms that care about the scholarly record as much as publishers. And sometimes it isn’t.

The Twitterverse

Lots of people use Twitter to talk about science. Some are scientists, some aren’t. Scientific articles are linked from news reports and discussed on blogs. Gone are the days of scholarly articles being cited only by other scholarly articles. We see links coming in from all over the place. And, although not all of this can be counted as the “scholarly record”, some of it could be.

The barrier-to-entry for journals publishing means that science journals contain only science articles. The barrier-to-entry for Twitter means that anyone can, and does, publish there. My Twitter feed is finely balanced between bibliometrics research, marine biology and pictures of snow leopards with Japanese captions. I don’t understand all of it, but I like looking at the pictures.

Back in the days when the only references to scholarly publications were from other scholarly publications, it was easy to keep track of those references. When an article was published, its references went into a citation database. This happened because the publisher considered this important.

But Twitter, the publisher of tweets, doesn’t care. It is used for a huge variety of communications and although some people choose to use it to engage in scholarship, we’re just a blip on their radar. The same goes for Reddit, a platform that describes itself as “the front page of the Internet”. There are communities engaged in scientific discussions, but Reddit doesn’t feel the need to publish its bibliographic references.

Nor should it.

Bridging those who care with those who don’t

The barrier-to-entry for contributing to scientific discussions has lowered, meaning that the role of more non-specialist platforms has increased.

I imagine that there are other communities out there who have their own concerns about the web. Maybe there are model train enthusiasts who want to keep track of every reference to a particular model. Or political commentators who want to keep track of how certain politicians and policies are discussed. As the scholarly community embraces new platforms for communicating, we should recognise that we are part of a broader universe of people using those platforms for more diverse reasons.

Gone are the days when the only way to reply to an article was by writing a letter to the editor. But also gone are the days when you could guarantee that your letter wouldn’t appear next to cat pictures (assuming you weren’t writing to the Journal of Feline Medicine & Surgery). As a specialist community cohabiting online spaces with non-specialists, it falls to us to do whatever we need to adapt that space and make it our own. In our case, this means recording bibliographic references as and where they occur.

Something like this happened once before. As traditional publishers went online, they created Crossref to build and maintain the necessary infrastructure. We’re acting on behalf of the community again to collect links from non-traditional sources. Because we can’t go to platforms like Twitter and say “please deposit your references”, we’re doing the opposite. We identify a platform, then work out how to scrape its content and extract links.

Working at scale

So we’re broadening out the universe of references that we would like to track from “traditional scholarly publishing” to “the entire web”. There are four broad challenges inherent in this, and we think that Crossref infrastructure is the right way to meet them.

The first challenge is physically finding the links. Because social media platforms aren’t specialised for scholarly publishing, they don’t have the same mechanisms in place for capturing bibliographic references. This means that we have to do it ourselves by scraping webpages for references. As the standard-bearer for scholarly PIDs, we think we can do a good job of this.

The second challenge is doing this at the scale of the web. Because we might, in theory, find a link on any webpage, there is a literally infinite number of publishing platforms. From big websites like BBC News down to tiny blogs run out of a bedroom. It would be impossible to partner with each of these individually. The way to solve this is to run a centralised service which goes out and contacts as many sources as possible. This role is a collaborative one. Our system is open to inspection, suggestions and contributions from the community.

The third challenge is the sheer number of publishers. Because they all register content with us, we are in good position to track their DOIs. In addition to that, every member of Crossref publishes content on their own platform, and has their own set of websites to track. We monitor our members’ websites and create a central list of domains that we look for. If this wasn’t done centrally, each publisher would have to run its own web crawlers and perform the same work, only to filter out their own links.

The fourth challenge is how to get all that data to the public. Even if every publisher were able to run their own infrastructure, it would make it very difficult to consume. Through Crossref metadata services, publishers have built a system where you can look up metadata and link to articles without worrying who published them. We think that the same approach should apply to this new link data.

For these reasons, we’re building Crossref Event Data: a system that monitors as many platforms as we can think of, and brings them into one place, and serves the whole community.

Building bridges

If you’ve been following along you’ll know that my last metaphor was the process of refining crude oil. I like metaphors, and mixing them. After all, you can’t mix a good metaphor without breaking a few eggs into the mixing bowl. Today’s metaphors are bridges. And not just one.

Bridge 1: PIDs and URLs

In the world of Persistent Identifiers, we’re quite good at linking. organisations like Crossref, DataCite and ORCID run separate systems but we work together to record and exchange links. But the web is different. There’s no single organisation in control and there are many organisations working to catalogue it. Event Data is our offering: bridging the web with our identifiers.

Bridge 2: Scholarly link providers

Of course, some platforms and systems do care about persistence and Persistent Identifiers. Event Data is an open platform, and we’re collaborating with a few providers to publish links.

We’ve partnered with The Lens to include Patent to DOI references. We’re working with F1000 to include links between reviews and articles. Hopefully we’ll see more organisations use Event Data to publish their links.

Bridge 3: Crossref / DataCite

Event Data is a collaborative project between DataCite and Crossref. When Crossref Registered Content contains a reference to a DataCite DOI we put it into Event Data. DataCite do the same in reverse. This means that Event Data contains a huge number of article - dataset links.

Bridge 4: Traditional discussions vs new ones

At each moment, scholarly discussions are happening in the literature, on various social media platforms and on the web at large. They are all talking about the same thing, but are spread out. Event Data collects links wherever we find them and brings them into one place. By doing this we hope we can help bring those conversations together.

Bridge 5: Bridging bibliometricians and altmetricians to data sources

Capturing links from social media to published literature underpins the field of altmetrics. By collecting this data and making it available under open licenses, we bring it to altmetrics researchers. We don’t provide metrics, but we do provide the data points that can form the basis for research.

Without infrastructure for collecting data, researchers would have to perform the same work over and over again. Because the data is all open, we allow datasets to be republished, reworked and replicated.

Bridge 6: Bridging the Evidence Gap

Running Event Data involves collecting a lot of data - gigabytes per day - and boiling it down into hundreds of thousands of individual Events per day. People consuming the data may want to do further boiling down. At every point of the process we record the input data that we were working from, the internal thought process of the system, and the Events that were produced. A researcher can use the Evidence Logs to trace through the entire process that led to an Event.

We’re a bridge from websites and social media to data consumers. But we take the role very seriously, and there’s nothing hidden. A glass bridge, you could say.

Interesting challenges

It’s not all plain sailing. There are a few challenges along the way to collecting this data which anyone who wanted to collect this kind of information would face. By collecting it in a central place and running an open platform we can solve each problem once, and improve our process as a community.

One problem is choosing what to include. We include any link that we find from a non-publisher website. That means that invariably some of the links are from spam. This problem isn’t new: we see low-quality articles being published in traditional journals from time to time. We try to include all of the data we can find and pass it onto consumers. They might want to whitelist certain sources, or they may want all of the data because they’re trying to study scholarly spam. We have decided to provide data as Events, which strike the balance between atomicity and usefulness.

Another, which I talked about at last year’s PIDapalooza, is how we track article landing pages. Read the blog post, the user guide or hop in a time machine if you’re interested.

The thing about bridges…

… is that they help people get where they’re going. With a few notable exceptions, they’re not the main attraction. We play a humble part in scholarly publishing, helping collect and distribute metadata. Most of what we do goes unseen, and helps people create tools, platforms and research. Event Data is an API, and whilst we hope people will build all kinds of things with it, including altmetrics tools, we’re not making another metric.

PIDapalooza

All of which brings me to my talk, which I’m giving on Wednesday: Bridging persistent and not-so-persistent identifiers. I would tell you about it, but there isn’t much more left to say.

If you want to find out more, we’re currently in Beta, and open for business. Head over to the User Guide to get started!

Event Data as Underlying Altmetrics Infrastructure at the 4:AM Altmetrics Conference

Joe Wass — Mon, 25 Sep 2017 00:00:00 +0000

I’m here in Toronto and looking forward to a busy week. Maddy Watson and I are in town for the 4:AM Altmetrics Conference, as well as the altmetrics17 workshop and Hack-day. I’ll be speaking at each, and for those of you who aren’t able to make it, I’ve combined both presentations into a handy blog post, which follows on from my last one.

But first, nothing beats a good demo. Take a look at our live stream. This shows the Events passing through Crossref Event Data, live, as they happen. You may need to wait a few seconds before you see anything.

Crossref and scholarly links

You may know about Crossref. If you don’t, we are a non-profit organisation that works with Publishers (getting on for nine thousand) to register scholarly publications, issue Persistent Identifiers (DOIs) and maintain the infrastructure required to keep them working. If you don’t know what a DOI is, it’s a link that looks like this:

https://doi-org.turing.library.northwestern.edu/10.5555/12345678

When you click on that, you’ll be taken to the landing page for that article. If the landing page moves, the DOI can be updated so you’re taken to the right place. This is why Crossref was created in the first place: to register Persistent Identifiers to combat link rot and to allow Publishers to work together and cite each other’s content. A DOI is a single, canonical identifier that can be used to refer to scholarly content.

Not only that, we combine that with metadata and links. Links to authors via ORCIDs, references and citations via DOIs, funding bodies and grant numbers, clinical trials… the list goes on. All of this data is provided by our members and most of it is made available via our free API.

Because we are the central place that publishers register their content, and we’ve got approaching 100 million items of Registered Content, we thought that we could also curate and collect altmetrics type data for our corpus of publications. After all, a reference from a Tweet to an article is a link, just like a citation between two articles is a link.

An Experiment

So, a few years back we thought we would try and track altmetrics for DOIs. This was done as a Crossref Labs experiment. We grabbed a copy of PLOS ALM (since renamed Lagotto), loaded a sample of DOIs into it and watched as it struggled to keep up.

It was a good experiment, as it showed that we weren’t asking exactly the right questions. There were a few things that didn’t quite fit. Firstly, it required every DOI to be loaded into it up-front, and, in some cases, for the article landing page for every DOI to be known. This doesn’t scale to tens of millions. Secondly, it had to scan over every DOI on a regular schedule and make an API query for each one. That doesn’t scale either. Thirdly, the kind of data it was requesting was usually in the form of a count. It asked the question:

“How many tweets are there for this article as of today?”

This fulfilled the original use case for PLOS ALM at PLOS. But when running it at Crossref, on behalf of every publisher out there, the results raised more questions than they answered. Which was good, because it was a Labs Experiment.

Asking the right question

The whole journey to Crossref Event Data has been a process of working out how to ask the right question. There are a number of ways in which “How many tweets are there for this article as of today?” isn’t the right question. It doesn’t answer:

Tweeted by who? What about bots?
Tweeted how? Original Tweets? Retweets?
What was tweeted? The DOI? The article landing page? Was there extra text?
When did the tweet occur?

We took one step closer toward the right question. Instead of asking “how many tweets for this article are there as of today” we asked:

“What activity is happening on Twitter concerning this article?”

If we record each activity we can include information that answers all of the above questions. So instead of collecting data like this:

Registered Content	Source	Count	Date
10.5555/12345678	twitter	20	2017-01-01
10.5555/87654321	twitter	5	2017-01-15
10.5555/12345678	twitter	23	2017-02-01

We’re collecting data like this:

Subject	Relation	Object	Source	Date
twitter.com/tweet/1234	references	10.5555/12345678	twitter	2017-01-01
twitter.com/tweet/5678	references	10.5555/987654321	twitter	2017-01-11
twitter.com/tweet/9123	references	10.5555/12345678	twitter	2017-02-06

Now we’re collecting individual links between tweets and DOIs, we’re closer to all the other kinds of links that we store. It’s like the “traditional” links that we already curate except:

It’s not provided by publishers, we have to go and collect it ourselves.
It comes from a very diverse range of places, e.g. Twitter, Wikipedia, Blogs, Reddit, random web pages
The places that the Events do come from don’t play by the normal rules. Web pages work differently to articles.

Non-traditional Publishing is Untraditional

This last point caused us to scratch our heads for a bit. We used to collect links within the ’traditional’ scholarly literature. Generally, journal articles:

get published once
have a publisher looking after them, who can produce structured metadata
are subject to a formal process of retractions or updates

Now we’re collecting links between things that aren’t seen as ’traditional’ scholarship and don’t play by the rules.

The first thing we found is that blog authors don’t reference the literature using DOIs. Instead they use article landing pages. This meant that we had to put in the work to collect links to article landing pages and turn them back into DOIs so that they can be referenced in a stable, link-rot-proof way.

When we looked at Wikipedia we noticed that, as pages are edited, references are added and removed all the time. If our data set reflected this, it would have to evolve over time, with items popping into existence and then vanishing again. This isn’t good.

Our position in the scholarly community is to provide data and infrastructure that others can use to create services, enrich and build things. Curating an ever changing data set, where things can disappear, is not a great idea and is hard to work with.

We realised that a plain old link store (also known as an assertion store, triple store, etc.) wasn’t the right approach as it didn’t capture the nuance in the data with sufficient transparency. At least, it didn’t tell the whole picture.

We settled on a new architecture, and Crossref Event Data as we now know it was born. Instead of a dataset that changes over time, we have a continual stream of Events, where each Event tells a new part of the story. An Event is true at the time it is published, but if we find new information we don’t edit Events, we add new ones.

An Event is the way that we tell you that we observed a link. It includes the link, in “subject - relation type - object” format, but so much more. We realised that one question won’t do, so Events now answer the following questions:

What links to what?
How was the link made? Was it with a article’s DOI or straight to an Article landing page?
Which Agent collected it?
Which data source were they looking at?
When was the link observed?
When do we think the link actually happened?
What algorithms were used to collect it?
How do you know?

I’ll come back to the “how do you know” a bit later.

What is an altmetrics Event?

So, an Event is a package that contains a link plus lots of extra information required to interpret and make sense of it. But how do we choose what comprises an Event?

An Event is created every time we notice an interaction between something we can observe out on the web and a piece of registered content. This simple description gives rise to some interesting quirks.

It means that every time we see a tweet that mentions an article, for example, we create an Event. If a tweet mentions two articles, there are two events. That means that “the number of Twitter events” is not the same as “the number of tweets”.

It means that every time we see a link to a piece of registered content in a webpage, we create an Event. The Event Data system currently tries to visit each webpage once, but we reserve the right to visit a webpage more than once. This means that the number of Events for a particular webpage doesn’t mean there are that many references.

We might go back and check a webpage in future to see if it still has the same links. If it does, we might generate a new set of Events to indicate that.

Because of the evolving nature of Wikipedia, we attempt to visit every page revision and document the links we find. This means that if an article has a very active edit history, and therefore a large number of edits, we will see repeated Events to the literature, once for every version of the page that makes references. So the number of Events in Wikipedia doesn’t mean the number of references.

An Event is created every time we notice an interaction. Each source (Reddit, Wikipedia, Twitter, blogs, the web at large) has different quirks, and you need to understand the underlying source in order to understand the Events.

We put the choice into your hands.

If you want to create a metric based on counting things, you have a lot of decisions to make. Do you care about bots? Do you care about citation rings? Do you care about retweets? Do you care about whether people use DOIs or article landing pages? Do you care what text people included in their tweet? The answer to each of these questions means that you’ll have to look at each data point and decide to put a weighting or score on it.

If you wanted to measure how blogged about a particular article was, you would have to look at the blogs to work out if they all had unique content. For example, Google’s Blogger platform can publish the same blog post under multiple domain names.

A blog full of link spam is still a blog. You may be doing a study into reputable blogs, so you may want to whitelist the set of domain names to exclude less reputable blogs. Or you may be doing a study into blog spam, so lower quality blogs is precisely what you’re interested in,

If you wanted to measure how discussed an article was on Reddit, you might want to go to the conversation and see if people were actually talking about it, or whether it was an empty discussion. You might want to look at the author of the post to see if they were a regular poster, whether they were a bot or an active member of the community.

If you wanted to measure how referenced an article was in Wikipedia, you might want to look at the history of each reference to see if it was deleted immediately. Or if it existed for 50% of the time, and to give a weighting.

We don’t do any scoring, we just record everything we observe. We know that everyone will have different needs, be producing different outcomes and use different methodologies. So it’s important that we tell you everything we know.

So that’s an Event. It’s not just a link, it’s the observation of a link, coupled with extra information to help you understand it.

How do you know?

But what if the Event isn’t enough? To come back to the earlier question, “how do you know?”

Events don’t exist in isolation. Data must be collected and processed. Each Agent in Crossref Event Data monitors a particular data source and feeds data into the system, which goes and retrieves webpages so it can make observations. Things can go wrong.

Any one of these things might prevent an Event from being collected:

We might not know about a particular DOI prefix immediately after it’s registered.
We might not know about a particular landing page domain for a new member immediately.
Article landing pages might not have the right metadata, so we can’t match them to DOIs.
Article landing pages might block the Crossref bot, so we can’t match DOIs.
Article landing pages might require cookies, or convoluted JavaScript, so the bot can’t get the content.
Blogs and webpages might require cookies or JavaScript to execute.
Blogs might block the Event Data bot.
A particular API might have been unavailable for a period of time.
We didn’t know about a particular blog newsfeed at the time.

This is a fact of life, and we can only operate on a best-effort basis. If we don’t have an Event, it doesn’t mean it didn’t happen.

This doesn’t mean that we just give up. Our system generates copious logs. It details every API call it made, the response it got, every scan it made, every URL it looked at. This amounts to about a gigabyte of data per day. If you want to find out why there was no Wikipedia data at a given point in time, you can go back to the log data and see what happened. If you want to see why there was no Event for an article by publisher X, you can look at the logs and see, for example, that Publisher X prevented the bot from visiting.

Every Event that does exist has a link to an Evidence Record, which corresponds with the logs. The Evidence Record tells you:

which version of the Agent was running
which Artifacts and versions it was working from
which API requests were made
which inputs looked like possible links
which matched or failed
which Events were generated

Artifacts are versioned files that contain information that Agents use. For example, there’s a list of domain names, a list of DOI prefixes, a list of blog feed urls, and so on. By indicating which version of these Artifacts were used, we can explain why we visited a certain domain and not another.

All the code is open source. The Evidence Record says which version of each Agent was running so you can see precisely which algorithms were used to generate the data.

Between the Events, Evidence Records, Evidence Logs, Artifacts and Open Source software, we can pinpoint precisely how the system behaved and why. If you have any questions about how a given Event was (or wasn’t) generated, every byte of explanation is freely available.

This forms our “Transparency first” idea. We start the whole process with an open Artifact Registry. Open source software then produces open Evidence Records. The Evidence Record is then consulted and turned into Events. All the while, copious logs are being generated. We’ve designed the system to be transparent, and for each step to be open to inspection.

We’re currently in Beta. We have over thirty million Events in our API, and they’re just waiting for you to use them!

Head over to the User Guide and get stuck in!

If you are in Toronto, come and say hi to Maddy or me.

You do want to see how it's made — seeing what goes into altmetrics

Joe Wass — Mon, 14 Aug 2017 00:00:00 +0000

There’s a saying about oil, something along the lines of “you really don’t want to see how it’s made”. And whilst I’m reluctant to draw too many parallels between the petrochemical industry and scholarly publishing, there are some interesting comparisons to be drawn.

Oil starts its life deep underground as an amorphous sticky substance. Prospectors must identify oil fields, drill, extract the oil and refine it. It finds its way into things as diverse as aspirin, paint and hammocks. And as I lie in my hammock watching paint dry, I’m curious to know how crude oil made its way into the aspirin that I’ve taken for the headache brought on by the paint fumes. Whilst it would be better if I did know how these things were made, not knowing doesn’t impair the efficacy of my aspirin.

Altmetrics start life deep inside a number of systems. Data buried in countless blogs, social media and web platforms must be identified, extracted and refined before it can be used in products like impact assessments, prompts to engagement, and even tenure decisions. But there the similarity ends. Like the benzene in my aspirin, the data that goes into my favourite metric has come a long way from its origins. But that doesn’t mean that I shouldn’t know how it was made. In fact, knowing what went into it can help me reason about it, explain it and even improve it.

Heavy industry or backyard refinery?

When you head out to fill your car, you buy fuel from a company that probably did the whole job itself. It found the crude oil, extracted it, refined it, transported it and pumped it into your car. Of course there are exceptions, but a lot of fuel is made by vertically integrated companies who do the whole job. And whilst there are research scientists who brew up special batches for one-off pieces of research, if you wanted to make a batch of fuel for yourself you’d have to set up your own back-yard fractional distillation column.

Because the collection of a huge amount of data must be boiled down into altmetrics, organisations who want to produce these metrics have a big job to do. They must find data sources, retrieve the data, process it and produce the end product. The foundation of altmetrics is the measurement of impact, and whilst the intermediary data is very interesting, the ultimate goal of a metric is the end product. If you wanted to make a new metric you’d have two choices: set up an oil refinery (i.e. build a whole new system, complete with processing pipeline) or a back-yard still (a one-off research item). Either option involves going out and querying different systems, processing the data and producing an output.

Being able to demonstrate the provenance of a given measurement is important because no measurement is perfect. It’s impossible to query every single extant source out there. And even if you could, it would be impossible to prove that you had. And even then, the process of refinement isn’t always faultless. Every measurement out there has a story behind it, and being able to tell that story is important when using the measurement for something important. Data sources and algorithms change over time, and comparing a year-old measurement to one made today might be difficult without knowing what underlying observations went into it. A solution to this is complete transparency about the source data, how it was processed, and how it relates to the output.

Underlying data

This is where Crossref comes in. It turns out that the underlying data that goes into altmetrics is just our kind of thing. As the DOI Registration Agency for scholarly literature, it’s our job to work with publishers to keep track of everything that’s published, assign DOIs and be the central collection and storage point for metadata and links. Examples of links stored in Crossref are between articles and funders, clinical trial numbers, preprints, datasets etc. With the Event Data project, we are now collecting links between places on the web and our registered content when they’re made via DOIs or article landing pages.

This data has wider use than just than altmetrics. For example, an author might want to know over what time period a link to their article was included in Wikipedia, and which edit to the article was responsible for removing it and why. Or, in these days of “fake news”, someone may want to know everywhere on Twitter that a particular study is referenced so they can engage in conversation.

Whilst the field of altmetrics was the starting point for this project, our goal isn’t to provide any kind of metric. Instead, we provide a stream of Events that occurred concerning a given piece of registered content with a DOI. If you want to build a metric out of it, you’re welcome to. There are a million different things you could build out of the data, and each will have a different methodology. By providing this underlying data set, we hope we’ve found the right level of abstraction to enable people to build a wide range of things.

Every different end-product will use different data and use different algorithms. By providing an open dataset at the right level of granularity, we allow the producers of these end-products to say exactly which input data they were working with. By making the data open, we allow anyone else to duplicate the data if they wish.

Sticky mess

To finish, let me return to the sticky mess of the distillation column. We identify sources (websites, APIs and RSS feeds). We visit each one, and collect data. We process that data into Events. And we provide Events via an API. At each stage of processing, we make the data open:

The Artifact Registry lists all of the sources, RSS feeds and domains we query.
The Evidence Registry lists which sites we visited, what input we got, what version of each Artifact was used, and which Events were produced.
The Evidence Log describes exactly what every part of the system did, including if it ran into problems along the way.
The Events link back to the Evidence so you can trace exactly what activity led up to the Event.
All the code is open source and the version is linked in the Evidence Record, so you can see precisely which algorithms were used to generate a given Event.
Anyone using the Data can link back to Events, which in turn link back to their Evidence.

The end-product, Events, can be used to answer altmetrics-y questions like “who tweeted my article?”. But the layers below that can be put to a range of other uses. For example:

“Why does publisher X have a lower Twitter count?”. The Evidence Logs might show that they tend to block bots from their site, preventing data from being collected.
“Why did their Twitter count rise?”. The Evidence Logs might show that they stopped blocking bots.
“What does Crossref think the DOI is for landing page X?”. A search of the Evidence Logs might show that the Event Data system visited the page on a given date and decided that it corresponded to DOI Y.
“Which domains hold DOI landing pages?”. The “Domains” Artifact will show the domains that Event Data looked at, and the Evidence Logs will show which versions were used over time.

By producing not only Events, but being completely transparent about the refinement process, we hope that people can build things beyond traditional altmetrics, and also make use of the intermediary products as well. And by using open licenses, we allow reuse of the data.

See you in Toronto!

There’s so much more to say but I’ve run out of ink. To find out more, come to 4:AM Altmetrics Conference! I’ll be speaking at the conference in Session 10 on the 28th. I’ll also be at the Altmetrics Workshop on the 26th. Stacy Konkiel and I are hosting the Hackathon on the 29th, where you can get your hands on the data. See you there!

This blog post was originally posted on the 4:AM Altmetrics Conference Blog.

Event Data enters Beta

Jennifer Kemp — Wed, 05 Jul 2017 00:00:00 +0000

We’ve been talking about it at events, blogging about it on our site, living it, breathing it, and even sometimes dreaming about it, and now we are delighted to announce that Crossref Event Data has entered Beta.

A collaborative initiative by Crossref and DataCite, Event Data offers transparency around the way interactions with scholarly research occur online, allowing you to discover where it’s bookmarked, linked, liked, shared, referenced, commented on etc., across the web, and beyond publisher platforms.

The name Event Data reflects the nature of the service, as it collects and stores digital actions that occur on the web, from the quick and simple, such as bookmarking and referencing, through to deeper interconnectivity such as exposing the links between research artifacts. Each individual action is timestamped and recorded in our system as an Event, and made available to the community via an API.

Event Data will be available for absolutely anyone to use; publishers, third party vendors, editors, bibliometricans, researchers, authors, funders etc., and with tens of thousands of events occurring every day, there’s a wealth of insight to be gained for those interested in analyzing and interpreting the data.

It’s important to note that Event Data does not provide metrics. What is does provide is the raw data to help you facilitate your own analysis, giving you the freedom to integrate the data into your own systems.

We are currently working very closely with a few organisations with specific use cases who are helping us to test and refine Beta before we launch our production service later this year. If you decide to take a look at Beta yourself, all the data you collect from Event Data is licensed for public sharing and reuse according to our Terms of Use.

Until Event Data is in production mode, we do not recommend building any commercial or customer-based tools off the data. If you are not in the Beta test group but are interested in participating, please contact me below. For more information about Event Data, please see our user guide.

Please contact me, Jennifer Kemp—Outreach Manager for Event Data—with any questions.