Using AI to Evaluate Internet Standards

One of the reasons I’m writing the Nature of Internet Standards series is to give policymakers tools that help them to understand and evaluate open standards efforts. A big part of that is understanding whether a specification (and the effort that led to it) is really a reflection of open deliberation, consensus processes, appropriate governance, and various other hallmarks of open standards.

Evaluating these factors requires a fair amount of context and nuance. What’s the most 2025 way to avoid that work? Using AI, of course.

Some colleagues were using AI to ask similar questions, and that inspired me to wonder whether it would be a useful option for policymakers who want to evaluate the openness of Internet standards. Let’s find out.

The Privacy Sandbox

Our first experiment is an effort that I’m fairly familiar with, having engaged with it both professionally and academically – Google’s Privacy Sandbox. How will AI portray this longstanding and very politicised effort?

Prompt

I started by asking ChatGPT’s “deep research” mode:

Is the Privacy Sandbox developed in an open standards process? Consider the venue(s) it’s developed in, their governance, openness, and conformance to other widely recognised signifiers of open standards processes, breadth of implementation, and adoption.

It asked for a few clarifications:

To give you the most accurate and up-to-date analysis, could you clarify a few things:

1. Are you primarily interested in the Privacy Sandbox on the web (i.e., as developed by Chrome and the W3C), or are you also considering the Android Privacy Sandbox?
2. Do you want a comparison to established open standards processes (e.g., W3C, IETF), or just an evaluation of the Privacy Sandbox initiatives on their own merits?
3. Are you looking for an overview of industry support (browsers, advertisers, publishers, etc.) and real-world implementation as part of your evaluation of adoption?

This will help me tailor the research more precisely.

To which I replied:

Just on the Web. A comparison to open standards processes would be good. I don’t need an overview of support and implementation, but the results should take these factors into account.

Response

After churning through data (and pushing the boundaries of copyright) for a couple of minutes, ChatGPT came up with this voluminous report. However, the heart of its response is concise and fairly damning:

In summary, the Privacy Sandbox has been developed in semi-open forums (W3C community groups, public repos, Google blogs) and invites outside input, but it is not a widely supported multi-vendor standard yet. Only Google’s Chrome (and related browsers) implements the APIs, and other browser vendors have developed their own approaches. The process resembles an open-source Google project more than a classic consensus-driven SDO. As Mozilla and others have noted, standards bodies like the W3C and IETF are the “ideal forum” to vet such proposals with full stakeholder oversight. At present, Privacy Sandbox’s governance does not fully align with those open standards processes – it provides considerable transparency and documentation, but its development and adoption remain largely controlled by Google.

That’s not far off. W3C Community Groups aren’t standards processes; in fact, they’re little more than mailing lists where the proponents get to set almost all of the rules. ChatGPT could have made this more clear, but instead uses lots of hedging phrases like “more than” and “fully align”.

Since the full report mentioned the chair of the Business Group that also discussed the Privacy Sandbox, Wendy Seltzer, I asked what she thought. Her response:

It’s fascinating to follow the sources here, because the Wikipedia article appears to have gotten its last substantive updates in 2024, and is mostly focused on older material ([which is] sometimes wrong). A human researcher would recognize that as not the best authority for recent news. A better human researcher would also ask “is this the question you want to ask, since the non-deprecation of third party cookies has changed the landscape around all these initiatives?”

That illuminates a broader issue of missing context. The answer starts from the premise that “Privacy Sandbox” is something that could be developed in an open standards process, rather than parsing the question, is PS more than the set of APIs and proposals that might fit into standards processes? How do open standards fit into the overall strategy? The GPT report can work for you, an expert in the field who can fill in those aspects, but it could be misleading to someone without the same background.

Some of the sentences quoted from other sources seem off without the context of those sources, e.g. “In January 2020 Google invited ad-tech companies to join the W3C’s Improving Web Advertising Business Group (IWABG) to discuss and give feedback on Privacy Sandbox ideas” reads as implying that Google was the only one doing the inviting.

QUIC

Next, let’s try another effort that I’m familiar with: QUIC, the effort to define a new, performance and security-focused transport protocol. I chaired the QUIC Working Group (along with Lars Eggert) from its inception until it shipped its core deliverables.

Prompt

I asked ChatGPT’s “deep research” mode a similar question for QUIC:

Is QUIC developed in an open standards process? Consider the venue(s) it’s developed in, their governance, openness, and conformance to other widely recognised signifiers of open standards processes, breadth of implementation, and adoption.

and once again, it asked for a similar clarification:

To best address your question about QUIC, could you please clarify what aspect you’re most interested in?

• Are you focused more on the technical development and governance process (e.g., how open the standardization process is)?
• Or are you more interested in its current implementation and adoption across the industry?
• Should I compare it to other protocols or standards in terms of openness and adoption?

Let me know so I can tailor the research accordingly.

I responded:

Focus on how open the standardisation process was, but consider implementation and adoption in your answer. A comparison with other relevant standards would be helpful.

Response

Here’s the full report. The summary is positively glowing:

In conclusion, QUIC’s standardization has been as open and transparent as any recent IETF protocol. It was developed in the open IETF process, adhering to OpenStand and WTO/TBT ideals of consensus, transparency, and voluntary adoption. Its final RFCs affirm community consensus, and the IETF infrastructure provided public visibility into every stage. This openness is mirrored by QUIC’s broad support: independent implementers from Google, Mozilla, Apple, Cloudflare, Facebook, and beyond have all deployed QUIC. The resulting ecosystem – with multiple browsers, CDNs, servers and apps supporting QUIC – reinforces that it is a truly open standard, not controlled by any single vendor. By contrast, older protocols like HTTP/2 and TCP underwent similar open processes (IETF consensus, public RFCs) and likewise enjoy ubiquitous implementation. Together, the history and adoption of QUIC demonstrate an effective, open standardization process in line with global best practices.

First of all, I love the reference to the “WTO/TBT ideals of consensus, transparency, and voluntary adoption.”

But is it accurate? Sort of. The facts here are correct: those companies did participate in the effort, implemented it, and deployed it. The process was open (as can be seen in the minutes and other materials).

I don’t know, however, how it justifies its statements about the quality of consensus – as I’ve written before, consensus can go wrong in a variety of ways. Others have characterised QUIC’s development has with Google firmly in the “driving seat.” Having been there for the whole standards process, I disagree with that assessment, but I’d at least expect academic work like that to be acknowledged.

Is AI Useful for Assessing Open Standards?

I can’t count the number of times that I’ve seen policymakers, journalists, and community members refer to the Privacy Sandbox as “at the W3C” or “being standardised at the W3C.” Given that extremely low bar, ChatGPT’s summary is an improvement. Likewise, I largely agree with its assessment of QUIC, at a high level.

What’s lacking here, however, is any kind of nuance. I can’t escape the feeling that it latches onto a few narratives that appear in source materials and augments them into well-worn clichés, like we see for QUIC. The IETF has a great reputation in many sources, so that gets amplified, but there’s a lack of any critical thought.

That’s not surprising: AI can’t think. If it could, it might wonder about the criteria we’re using for “open standards” here – are those WTO/TBT ideals still relevant, and are they adequately described? Are the processes actually used in working groups lining up with the rhetoric of openness – and how would you find out if they didn’t? And, how much should all of that count if the result isn’t proven by market adoption?

In a nutshell: if you must use AI to assess the openness of a standard, only use it for the first pass, check all of the references, and then roll up your sleeves and start talking to people to get the real story.