Current AI Language Models Fall Short of EU AI Legislation Requirements

Stanford Study Reveals Non-Compliance of AI Tools

Researchers at Stanford University recently discovered that present-day large language models (LLMs), including AI technologies like OpenAI’s GPT-4 and Google’s Bard, do not conform to the regulations set forth by the European Union (EU) Artificial Intelligence (AI) Act.

The EU AI Act: A Groundbreaking Legislation for AI

This Act, a groundbreaking legislation that seeks to govern AI on both a national and regional scale, has been recently ratified by the European Parliament. It sets rules for AI within the EU, influencing a population of around 450 million, and stands as an avant-garde model for AI legislation globally.

However, the Stanford study suggests that for AI corporations to become compliant, they face a challenging journey ahead.

Assessment of Compliance with the AI Act

In their research, the Stanford team examined ten leading model providers, scrutinizing their compliance level with the 12 criteria stated in the AI Act, on a scale of 0 to 4.

The study found a stark disparity in levels of compliance among providers. Some scored less than 25% on compliance with the Act’s stipulations, with only one provider, Hugging Face/BigScience, achieving a score above 75%.

Evidently, even the providers with high scores have substantial room for enhancement.

Key Non-Compliance Issues and the Need for Improvement

The research highlighted key areas of non-compliance, including a transparency deficiency in reporting the status of copyrighted training data, energy consumption, emissions, and the strategy to manage potential risks.

In addition, a noticeable contrast was observed between open and closed model releases, where open releases offered a more comprehensive disclosure of resources, but presented greater challenges in monitoring and controlling deployment.

The Stanford team posited that all providers, irrespective of their release strategy, have room for substantial improvements.

Diminishing Transparency and the Complex Relation with Regulatory Bodies

In recent times, there has been a significant decrease in transparency with major model releases. For instance, OpenAI failed to disclose data and compute information for GPT-4, attributing it to competitive reasons and safety considerations.

These observations are part of a broader trend. OpenAI has recently been advocating for changes in countries’ stance towards AI, even suggesting a departure from Europe if regulations were overly restrictive, a statement they later withdrew. These actions highlight the often contentious relationship between AI providers and regulators.

Recommendations for Regulatory Improvement

The Stanford team proposed several measures to enhance AI regulation. They suggested that EU legislators should ensure that the AI Act holds larger model providers accountable for transparency and accountability. They also underscored the need for technological expertise and talent to implement the Act effectively, given the complexity of the AI landscape.

The researchers believe that the primary hurdle is the speed at which model providers can adapt their business models to comply with the regulatory requirements. They noted that without substantial regulatory pressure, many providers could achieve high scores (in the 30s or 40s out of 48) with meaningful but feasible changes.

The Future of AI Regulation

The Stanford study provides a glimpse into the future of AI regulation, asserting that the AI Act, if enacted and enforced, will create a significant positive impact on the AI landscape, fostering transparency and accountability.

AI is revolutionizing society with its unparalleled abilities and inherent risks. As we stand on the brink of regulating this transformative technology, the critical role of transparency is coming to the fore, not just as an optional component, but as a fundamental requirement for responsible AI deployment.