mediaeval2026-sid

MediaEval 2026 Synthetic Images Detection Challenge: Advancing detection of generative AI used in real-world online images

This repository contains the data and submission instructions for the MediaEval 2026 tasks on Synthetic Image Detection and Manipulated Region Localization.

See the MediaEval 2026 webpage for information on how to register and participate.

Details about submission procedures will be communicated separately.

---

📌 Task A: Real vs. Synthetic Image Detection

📝 Task Description

Participants are given a curated set of 5,000 real and 5,000 synthetic images collected from in-the-wild sources. The challenge is to develop models that classify whether a given image is real or synthetic.

The task is structured into two required runs:

• Constrained Run: Use only the official training dataset. Details on this dataset are included in the Data Description section.
• Open Run: Use any data, including external or self-generated synthetic images.

The goal is to build and evaluate models that can accurately detect whether a given image is synthetic or real, especially under the challenging conditions posed by in-the-wild images.

---

📂 Data Description

✅ Training Data (for Constrained Run)

• Wang et al. (2020) A benchmark dataset of synthetic images generated by GAN-based models (StyleGAN2, BigGAN, ProGAN). These models are now considered relatively obsolete but are still widely used for benchmarking in SID research. The dataset can be found here.

• Corvi et al. (2022) A more recent dataset of 200K synthetic images (Latent Diffusion) and 200K real images (COCO, SUN datasets). The dataset can be found here.

✅ Validation and Testing Data

• 10k labeled images for validation (5k real, 5k synthetic)

Participants are provided with a curated set of real and synthetic images collected in the wild to validate and evaluate their models. The images are organized into two subdirectories:

0_real/    # 5k real images  
1_fake/    # 5k synthetic images  

Validation dataset can be downloaded here.

• 10k images for testing

The test set consists of 10k images collected from social media platforms, similar in nature to the images in the validation set.

⚠️ The test dataset will be sent ONLY to registered participants who have signed and returned the usage agreement.

---

📤 Expected Submission Format

Each run must be submitted as a CSV file.

Submit two runs, each as a single .csv file:

• teamname_constrained.csv
• teamname_open.csv

Each CSV must contain:

Column	Description
image_id	Filename of the image (e.g., fH5gBIhTRXWW.jpg)
prob	Probability that the image is synthetic (0.0–1.0 float)
label	Binary prediction (0 = real, 1 = synthetic)
threshold	Global threshold used for classification (e.g., 0.5)

The CSV file must include a header row with the column names in order.

🧪 Example:

image_id,prob,label,threshold  
image_000123.jpg,0.876,1,0.5

This indicates that the model predicted an 87.6% probability of the image being synthetic, applied a threshold of 0.5, and therefore classified the image as synthetic (label=1).

⚠️ Use a single global threshold for each run, and repeat it in each row.

⚠️ Participants must ensure that their CSV files include predictions for all images in the test set, and that file naming strictly matches the provided image filenames. Do not submit both run types in a single file. Submissions that do not follow this format may be disqualified from evaluation.

---

🧭 Task B: Manipulated Region Localization

📝 Task Description

This task involves:

1. Predicting whether an image has been synthetically manipulated.
2. Producing a pixel-level mask identifying manipulated regions.

---

📂 Data Description

✅ Training Data

Participants are required to use the TGIF dataset (Mareen et al., 2024) and SAGI (Giakoumoglou et al., 2025) for training, and may use these datasets for validation.

The TGIF dataset is split into two types of manipulated images: spliced (sp), where the inpainted area is integrated in the original image and fully regenerated (fr), where the entire image is synthetically regenerated despite only the inpainted region being semantically manipulated. The spliced images were made using Adobe Photoshop (ps) and Stable Diffusion 2 (sd2), whereas the fully regenerated images were made using Stable Diffusion 2 and Stable Diffusion XL (sdxl).

More details about the dataset can be found here, while the data can be downloaded from this link.

The SAGI dataset is a comprehensive dataset of AI-generated image inpaintings, containing over 95,000 high-quality inpainted images created using state-of-the-art models. This dataset is suitable for training inpainting detection algorithms as it provides the inpainted images and their respective inpainting masks and source images. The dataset includes in-domain evaluation sets with source images from COCO and RAISE datasets with captions generated by ChatGPT, as well as an out-of-domain test set using OpenImages with captions from Claude Sonnet, enabling assessment of detection algorithm performance across different data distributions and captioning sources.

More details about the dataset can be found here, while the data can be downloaded from this link.

Participants are free to use all splits of the TGIF/SAGI-D datasets in any manner they consider appropriate for development.

✅ Validation Data

The validation dataset is structured into two main source directories: coco and raise.

COCO includes an original subdirectory containing 1,950 unaltered images while raise contains 1277 original images. In addition for both sources, there are seven subdirectories corresponding to different manipulation methods.

• brushnet, controlnet, hdpainter, inpaintanything, mixed, powerpaint, removeanything

Each of these manipulation method folders contains two subfolders: image/, which holds the manipulated versions of the original images, and mask/, which contains the corresponding binary masks in PNG format, highlighting the altered regions. These validation images have been constructed to closely resemble the types of manipulated content expected in the test set, making them more representative of real-world scenarios and better suited for evaluating model generalization.

For both COCO and RAISE, the structure is:

coco/
 └── original/          # 1,950 authentic images
 └── brushnet/
     └── image/         # manipulated images
     └── mask/          # binary PNG masks
... (same structure for controlnet, hdpainter, etc.)

The validation dataset can be found here.

A separete CSV file provides detailed information for each image in the validation set, including the manipulation method and the image generation strategy.

• The type column indicates the nature of the manipulation:
  • "sp" (spliced): The region outside the inpainting mask has been preserved using post-processing techniques.
  • "fr" (fully regenerated): The entire image, including regions outside the inpainting mask, has been regenerated.

📄 Download validation details CSV

The test dataset is of the same nature as the validation dataset. It contains original images sourced from COCO, RAISE, and OpenImages, and the same manipulation methods have been applied to generate the altered versions. Participants are provided with all test images, both original and manipulated, without any labels or additional metadata.

⚠️ The test dataset will be sent ONLY to registered participants who have signed and returned the usage agreement.

---

📤 Expected Submission Format

1. Masks (.npz)

• One .npz file per image in the test set.
• Each file contains a (H, W) float16 array of probabilities (0.0 to 1.0).
• Filename must match the image, e.g. fEdOddAW3EeT.npz for fEdOddAW3EeT.jpg.

All .npz files must be packaged into a single .zip archive for submission. Name your file as:

teamname_localization_masks.zip

⚠️ Participants must provide a mask file for every test image, including those where the model fails to detect any manipulated region. In such cases, the mask should be a valid array of the same size as the input image, containing only 0.0 values (i.e., zero probability across all pixels).

2. Global Scores (scores.csv)

In addition to the mask files, participants must submit a CSV file named scores.csv, following the same structure as in the classification task. The CSV must include five columns:

Column	Description
image_id	Filename of the image
prob	Global manipulation probability (float between 0–1)
label	Binary label (0 = real, 1 = manipulated)
threshold	Global threshold used for classification
loc_threshold	Threahold that can be used with mask probabilities

🧪 Example:

image_id,prob,label,threshold,loc_threshold  
image_000123.jpg,0.715,1,0.5,0.5

---

Quest for insight

We encourage the participants to go beyond just looking at the evaluation metrics. As a starting point, here are several research questions:

• Are the synthetic images and image regions that are challenging to detect automatically also challenging for people looking at the images to see?
• What are the characteristics of the false positives, i.e., of non-synthetic images or image regions that are misclassified as synthetic? Are there any general insights that can be drawn in terms of perceived quality or image semantics?
• Are there characteristics (semantic content, framing) that are specific to the in-the-wild examples?
• How does the choice of training data influence model performance and behavior?
• What benefits arise from creating or sourcing new datasets? For example, from fact-checking platforms or social media? Which types of content remain underrepresented?
• How close can collected data come to the distribution of real online synthetic media, and what strategies help reduce the mismatch between curated training sets and real-world content?

---

📅 Important Dates

Event	Date
Data release	March 1
Runs due	May 1 -> May 15
Paper submission	May 31
Workshop	June 15–16

📍 Attendance to the MediaEval Workshop is required (online or in person).

---

📬 Contact Information

For more details about the challenge, please contact:

• Olga Papadopoulou – olgapapa@iti.gr, MeVer group, CERTH-ITI, Greece

---

Citation

If you participate in the task and/or use the dataset, baseline, or task materials, please cite the following papers:

@inproceedings{papadopoulou2026synthetic,
  title     = {Synthetic Images at MediaEval 2026: Advancing Detection of Generative AI in Real-World Online Images},
  author    = {Papadopoulou, Olga and Karageorgiou, Dimitrios and Koutlis, Christos and Gavves, Efstratios and Mareen, Hannes and Papadopoulos, Symeon},
  booktitle = {Proceedings of MediaEval'26: Multimedia Evaluation Workshop},
  year      = {2026},
  address   = {Amsterdam, Netherlands and Online},
  eventdate = {2026-06-15/2026-06-16}
}

@inproceedings{papadopoulou2025synthetic,
  title     = {Synthetic Images at MediaEval 2025: Advancing Detection of Generative AI in Real-World Online Images},
  author    = {Papadopoulou, Olga and Schinas, Marios and Corvi, Riccardo and Karageorgiou, Dimitrios and Koutlis, Christos and Guillaro, Fabrizio and Gavves, Efstratios and Mareen, Hannes and Verdoliva, Luisa and Papadopoulos, Symeon},
  booktitle = {Proceedings of the MediaEval 2025 Workshop},
  year      = {2025},
  pages     = {25--26},
  address   = {Dublin, Ireland and Online}
}