AI-Generated Images Often Can't Be Traced Back to Training Data, Study Finds

A new study from researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) has shed light on the complex issue of attribution in AI-generated images. The team found that for models trained on large datasets, it may often be impossible to determine which specific training examples contributed to a particular output. This phenomenon is known as attribution decay, where individual pieces of data become increasingly irrelevant at sufficiently large scales.

The researchers identified this problem through an innovative approach called ablation, which involves removing individual pieces of data from the training set and observing whether it affects the model’s output. To their surprise, they found that even when a single image was removed, the generated sample often didn’t change significantly. This suggests that attribution is not as straightforward as previously thought.

The team developed an architecture called a ‘diffusion ensemble’ to overcome this challenge. Instead of relying on one monolithic model, it’s composed of multiple smaller components, each trained on different slices of data. By switching off the parts that saw a particular image, they can generate true counterfactual models without retraining or approximation.

The diffusion ensembles were put head-to-head with 24 conventional diffusion models trained on the same data, and the results showed no significant difference in terms of image quality by standard measures. However, one notable finding was that the more training data a model had access to, the better it performed compared to its single-model counterparts.

The researchers then explored the concept of an ‘image’s counterfactual universe’, which refers to all possible alternate versions of a generated image produced by removing different pieces of training data. They found that as the size of the training set increased, the distance between the original and its most dissimilar alternate decreased, following an inverse power law.

The study also involved stress-testing their results through various methods, including retraining models from scratch without specific images and pinning the removed fraction in place to see if it persisted. The findings held up across different metrics and conditions, suggesting that attribution decay is a real phenomenon that affects AI-generated images at large scales.

One of the key implications of this research is its potential impact on copyright law and fair use. If model outputs are not attributable to individual pieces of training data, does that mean they can be considered novel works? The study’s findings raise questions about how authors get compensated when what comes out of a model isn’t tied to anything online.

The researchers also note that their work shows how to produce outputs guaranteed to be unattributable. This capability is framed as an obligation for the industry, rather than a loophole. Companies would need to revise their models to take advantage of these advances in order to claim their outputs aren’t derivative works.

While this study focused on diffusion models, which are currently dominant in generating audiovisual media and prevalent in scientific applications, it’s unclear whether similar decay occurs with large language models at the center of high-profile copyright litigation. The researchers suggest that attribution may fail for interesting models, requiring technologists and courts to resort to other methods for assessing copying.

The study was supported by Schmidt Futures, and its findings have significant implications for AI-generated images, data analysis tools, and businesses relying on these technologies. As the use of AI continues to grow in various industries, understanding how attribution works – or doesn’t work – is crucial for ensuring responsible development and deployment.