Index Reingest Complete!

[pgulley]

Last month, we finally completed our years long journey to reindex all of our archived historical content into the Media Cloud index! This is a major milestone which signals the end of a chapter of reconstruction of our codebase and dataset.

I'd like to share a preview of a chart we'll be publishing which shows a birds eye view of the data in the index. This approximately [1] plots the story volume over time, grouped into strata based on the earliest appearance of a given source in our index. Ie- sources that were a part of the original seed sources in 2008 contribute to the baby-blue band at the bottom, and sources added in 2013 contribute to the orange band in the middle.

The earliest consistent ingest processes for this project began in 2008, with a relatively small scale operation. Since then, over multiple generations of contributors and technical approaches, we've scaled up to the point where we now ingest content from more than a hundred thousand RSS feeds (you can monitor the continuous ingest process here ) . Our current pool of core contributors (excluding PIs on the project) has only been working on the system since 2022, so our immediate focus since finishing re ingestion has been to try to make sense of the rocky-road that precedes our tenure. We anticipate that most of the gaps are products of the techniques we used to fill in data which was corrupted or missing from our archive, but there are some mysteries remaining. Look forward to a blog post from the team sharing our findings soon.

Huge thanks to the developers who helped us get here, especially @philbudne, @rahulbot, @kilemensi, @thepsalmist, @m453h

[1] - A Nota Bene about this chart is that it's only aggregated over the top 100 sources in each individual time-frame, so the overall volume of each band appears to shrink as the total diversity of the index increases- this is a counting artifact that will be addressed in future work.

[rahulbot]

Great to understand how this evolved 👏🏽 Two notes of historical context:

• I think the orange 2013 bump is when we added collections from the European Media Monitor. A search against that collection confirms it as an early 2013 addition.
• older GH issues show we added a bit set of global sources from ABYZ news links in Dec 2017. I think that's the purple band.