The Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) is a low-barrier mechanism for repository interoperability. Data Providers are repositories that expose structured metadata via OAI-PMH. Service Providers then make OAI-PMH service requests to harvest that metadata. -- https://www.openarchives.org/pmh/
Note to existing users - two significant changes starting with 0.5.0:
- we use a single command
methaand subcommands for the previously separate binaryies, e.g.metha-syncbecomesmetha sync(we provide shims for the transition) - we switch to a new internal storage layout: to migrate an existing cache to the new layout, simply type:
$ metha migrate --dry-run
This will list the cached endpoint data, that will be converted. If that looks good, run:
$ metha migrate --rm # to remove the previous files after migration is fully done
The metha command line tool can gather information on OAI-PMH endpoints and harvest data incrementally. The goal of metha is to simplify data access.
The metha tool has been developed for project finc at Leipzig University Library (lab).
- I wanted to crawl Arxiv but found that existing tools would timeout.
- Some harvesters would start to download all records anew, if I interrupted a running harvest.
- There are many OAI endpoints out there. It is a widely used protocol and somewhat worth knowing.
- I wanted something simple for the command line; also fast and robust, while not stressing servers too much
The functionality is spread accross a few different subcommands:
- metha sync for harvesting
- metha cat for viewing
- metha id for gathering data about endpoints
- metha ls for inspecting the local cache
- metha files for listing the associated files for a harvest
- metha sweep for harvesting every known endpoint, on a schedule
- metha endpoints for what the sweep learned about each one
- metha export for writing the whole cache out as one stream of records
To harvest and endpoint in the default oai_dc format (e.g. arxiv.org):
$ metha sync https://oaipmh.arxiv.org/oai
...All downloaded files are written to a directory below a base directory. The
base directory is ~/.cache/metha by default ("xdg") and can also be adjusted
with the METHA_DIR environment variable.
When the --dir flag is set, only the directory corresponding to a harvest is printed.
$ metha sync --dir https://oaipmh.arxiv.org/oai
/Users/tir/Library/Caches/metha/62/4a/624afaf261ab83ec
$ METHA_DIR=/tmp/ ./metha sync --dir https://oaipmh.arxiv.org/oai
/tmp/62/4a/624afaf261ab83ecThe harvesting can be interrupted at any time and the HTTP client will automatically retry failed requests a few times before giving up.
To stream the harvested XML data to stdout run:
$ metha cat https://oaipmh.arxiv.org/oaiYou can emit records based on datestamp as well:
$ metha cat --from 2016-01-01 https://oaipmh.arxiv.org/oaiThis will only stream records with a datestamp equal or after 2016-01-01.
To display basic repository information:
$ metha id https://oaipmh.arxiv.org/oaiTo list all harvested endpoints on your machine:
$ metha lsFurther examples can be found in the metha man page:
$ man metha # when installed from a package
Use a deb, rpm release, or the go tool:
$ go install -v github.com/miku/metha/cmd/metha@latestSince 0.5 metha is a single binary with one subcommand per verb: metha sync,
metha cat, metha ls and so on. See metha help.
The nine commands metha used to install — metha-sync, metha-cat, ... are
still there as symlinks to it. metha reads the name it was invoked under and
runs the matching verb, so existing scripts keep working, flags included. The
packages install those names for you; after a go install you can add them with
$ metha shim installThey print a one-line deprecation notice when run from a terminal (silence it
with METHA_NO_DEPRECATION=1) and will be removed in metha 2.0.
go install github.com/miku/metha/cmd/metha-sync@latest and friends still
resolve throughout the 0.5.x line, but each one builds the whole program: nine
of them come to about 186MB, where the single binary is about 25MB.
In 0.1.27 a metha fortune command was added, which fetches a random article
description and displays it.
$ metha fortune
Active Networking is concerned with the rapid definition and deployment of
innovative, but reliable and robust, networking services. Towards this end we
have developed a composite protocol and networking services architecture that
encourages re-use of protocol functions, is well defined, and facilitates
automatic checking of interfaces and protocol component properties. The
architecture has been used to implement common Internet protocols and services.
We will report on this work at the workshop.
-- http://drops.dagstuhl.de/opus/phpoai/oai2.php
$ metha fortune
In this paper we show that the Lempert property (i.e., the equality between the
Lempert function and the Carathéodory distance) holds in the tetrablock, a
bounded hyperconvex domain which is not biholomorphic to a convex domain. The
question whether such an equality holds was posed by Abouhajar et al. in J.
Geom. Anal. 17(4), 717–750 (2007).
-- http://ruj.uj.edu.pl/oai/request
$ metha fortune
I argue that Gödel's incompleteness theorem is much easier to understand when
thought of in terms of computers, and describe the writing of a computer
program which generates the undecidable Gödel sentence.
-- http://quantropy.org/cgi/oai2
$ metha fortune
Nigeria, a country in West Africa, sits on the Atlantic coast with a land area
of approximately 90 million hectares and a population of more than 140 million
people. The southern part of the country falls within the tropical rainforest
which has now been largely depleted and is in dire need of reforestation. About
10 percent of the land area was constituted into forest reserves for purposes
of conservation but this has suffered perturbations over the years to the
extent that what remains of the constituted forest reserves currently is less
than 4 percent of the country land area. As at today about 382,000 ha have been
reforested with indigenous and exotic species representing about 4 percent of
the remaining forest estate. Regrettably, funding of the Forestry sector in
Nigeria has been critically low, rendering reforestation programme near
impossible, especially in the last two decades. To revive the forestry sector
government at all levels must re-strategize and involve the local communities
as co-managers of the forest estates in order to create mutual dependence and
interaction in resource conservation.
-- http://journal.reforestationchallenges.org/index.php/REFOR/oaimetha sweep harvests every endpoint metha knows about — 244,040 of them,
seeded from the embedded list — records what became of each one, and exits.
Previously conducted example scrapes, converted to JSON: 326M records, 60+ GB:
2023-11-01-metha-oai.ndjson.zst,
and
2026-02-23-oaiscrape-unique.jsonl.zst
(214M records, 41GB compressed).
$ metha sweep --dry-run # what is due, without a single request
$ metha sweep --limit 100 # try it on a hundred endpoints
$ metha sweep # everything due, with the defaultsIt keeps a roster beside the cache, sweep.json.zst, holding one profile per
endpoint: when it was last attempted, what happened, and when it is next due.
An endpoint that answers is polled daily; one that has never resolved backs off
to a few requests a year (and is never dropped, because repositories move and
domains come back). Requests are partitioned by host, so a repository with
several hundred endpoints is never asked more than one question at a time.
A sweep is bounded twice: --deadline per endpoint (1h), --budget for the
whole run (24h) and everything harvested before either fires is kept. Two
sweeps cannot overlap: the second finds the lock held, says so, and exits 0.
metha endpoints is the view onto what it learned, and it prints URLs one per
line so its output is an input:
$ metha endpoints --state quarantined # what has stopped answering
$ metha endpoints --class gone # what never answered at all
$ metha endpoints --slower-than 5m --json # what a sweep spends its time on
$ metha endpoints --state active # the corrected endpoint list
$ metha endpoints --import my-endpoints.txt # add your ownTo run it nightly, use the metha.service and metha.timer units; see extra/linux for how to install them, and for what they replaced.
metha stores harvested data in one file per interval. metha export writes all
of it out as one stream, one JSON document per line:
$ metha export -o corpus.ndjson.zst # compressed by extension
$ metha export | jq .header.identifier # or straight down a pipe
$ metha export --from 2024-01-01 # only what is recentEvery line carries an endpoint field naming the repository the record came
from; the one thing an OAI-PMH record does not say about itself, and the one
thing a corpus of a few hundred million of them needs:
$ metha export | jq -r '[.endpoint, .header.identifier] | @tsv'It reads and never writes the cache, and takes no locks, so it is safe to run
while a sweep is harvesting. To export part of the corpus, name endpoints as
arguments or pass a file of them (which is what metha endpoints prints):
$ metha endpoints --state active > live.txt
$ metha export --endpoints live.txt -o live.ndjson.zst--xml writes one XML document instead, and the record filters metha cat
has: --from, --until, --setspec, --deleted work the same way here.
- responses with resumption tokens that lead to empty responses
- gzipped responses, that are not advertised as such
- funny (illegal) control characters in XML responses
- repositories, that won't respond unless the dates are given with the exact granualarity
- repositories with endless token loops
- repositories that do not support selective harvesting, use
-no-intervalsflag - limited repositories, metha will try a few times with an exponential backoff
- repositories, which throw occasional HTTP errors, although most of the responses look good, use
-ignore-http-errorsflag
- Martin Czygan martin.czygan@uni-leipzig.de
- Natanael Arndt, https://github.com/white-gecko
- Gunnar Þór Magnússon, https://github.com/gunnihinn
- Thomas Gersch, https://github.com/titabo2k
- Andreas Czerniak
- David Glück
- Justin Kelly
- Claude Opus 4.8/5, ZAI GLM 5.2/3, ...
Show formats of random repository:
$ shuf -n 1 <(curl -Lsf https://git.io/vKXFv) | xargs -I {} metha id {} | jq .formatsA snippet from a 2010 publication:
The Open Archives Protocol for Metadata Harvesting (OAI-PMH) (Lagoze and van de Sompel, 2002) is currently implemented by more than 1,700 digital library reposi- tories world-wide and enables the exchange of metadata via HTTP. -- Interweaving OAI-PMH Data Sources with the Linked Data Cloud
- The finc project
- UB LEIPZIG LAB
- Getting a dump of arXiv metadata at academia.stackexchange.com
- Keyword Extraction from arXiv - Part 1
- Openrefine use case: Automated workflow for harvesting, transforming and indexing of bibliographic metadata
- Sammeln und Finden. Über das Sichtbarmachen von Open Science in Hamburg (PDF)
- acohan/arxiv-tools
- Arxiv on Archive
- Metadata analysis of 80,000 arxiv:physics/astro-ph articles
- Arxiv Harvesting
- Orcid
- Tutorial on Topological Data Analysis
- Connectome (linked open research data, #28), From MARCXML to Records in Contexts
- Comparison w/ other OAI tools (de)
- biblio.ai
- Datasets ("oaiscrape") generated by metha has been used for Preserving Scholarly Communications on the Web with Open Metadata (2025)
