A recent surge in advocacy for JSON-formatted prompts has swept through social media, particularly among developer circles in the AI space. These posts, usually in threads that begin with titles such as “You’re missing out big time if you’re not using JSON for LLMs”, often promise better outputs and model performance.
However, seasoned developers are pushing back. Noah MacCallum, who works with the applied AI team at OpenAI, didn’t mince his words, addressing one such post on the JSON fanfare.
“Guys, I hate to rain on this parade, but JSON prompting isn’t better,” he said, indicating that it’s just ‘hype’ and it ‘physically pains’ him that such posts are getting traction. “I’ve actually done experiments on this, and Markdown, or XML, is better,” he added.
MacCallum’s criticisms focus on three fundamental issues with using JSON for prompting. One is the trading data fallacy and the popular argument that models are trained on JSON, therefore respond better. However, MacCallum points out that they’re also trained on a massive amount of plain text, Markdown formats and more. Thus, the mere presence of JSON in training data doesn’t make it the optimal prompting format.
Besides, he also said that JSON isn’t token efficient and creates tonnes of noise and attention load with the whitespace, escaping, and keeping track of closing characters. Moreover, in a scenario where token costs and context limits matter, this inefficiency compounds quickly.
Perhaps most critically, MacCallum stated that using JSON causes the model to behave as if it is reading and has to output code, paying less attention to what the user may genuinely want.
OpenAI’s official documentation also recommends using Markdown or XML, while noting that JSON performed ‘poorly’ compared to XML for long context inputs.
Even OpenAI’s new open-source GPT-OSS model requires what is called a “Harmony” response format, featuring XML-like tags for structured conversations, reasoning channels, and tool calls. This format enables multi-channel outputs including analysis, commentary, and final responses, reinforcing the industry trend towards XML over JSON for LLM prompting.
Moreover, Anthropic also suggests using XML format, even calling it a ‘game changer’.
Anthropic states that having Claude use XML tags in its output facilitates the extraction of specific parts of its response during post-processing, and XML also helps the user easily add, remove, or modify parts of a prompt without rewriting.
‘Just Use Clean, Well-Formatted Markdown or Text’
Developers AIM contacted either supported the observations above or proposed a different approach.
Jared Zoneraich, founder of PromptLayer, told AIM he advocates for a context-switching heuristic when working with LLMs. He treats these models like humans who have difficulty switching between different conversational modes.
Since LLMs operate as pattern-matching systems, asking for creative output while using JSON formatting creates a mismatch — the JSON structure primes the model to match against technical patterns rather than creative ones, ultimately degrading the quality of the desired creative response.
Zoneraich said that he uses XML, or Markdown when he prompts. “Many models are trained on XML, Markdown, or, often, both. Claude is famously good at XML. I often use Markdown for sections and for ‘human readability’ and XML to denote ‘attachments’,” he added.
With that being said, one independent developer who works on LLM quantisations, John Leimgruber, had a contrasting view to share. He told AIM, “I have no idea why anyone would use XML or JSON prompting for LLMs. I always use Markdown.”
“Markdown is the ‘lingua franca’ of LLMs and developers for years now,” he added.
He said that while some LLMs do use what appears to be XML-style tags, it is more of a training/encoding formality, but not a suggestion to use XML in your own prompts.
“Just use clean, well-formatted simple markdown. There is no reason to do otherwise,” he emphasised, stating that using JSON or XML would add a lot of extra characters to the prompt, and would cost more or take longer to process.
Similarly, even OpenAI states that “The model [GPT 4.1, in context] is trained to robustly understand structure in a variety of formats. Generally, use your judgement and think about what will provide clear information and ‘stand out’ to the model. For example, if you’re retrieving documents that contain lots of XML, an XML-based delimiter will likely be less effective.”
Leimgruber also touched upon a less discussed narrative. He noted that system prompts used by leading AI labs to instruct their models are all based on plain text or markdown-formatted text. “They don’t use JSON or XML-formatted system prompts internally,” he said.
For example, the recent addition to ChatGPT, the ‘Study Mode’, involved OpenAI providing a system prompt based on plain text, containing ‘strict’ rules that instructed the AI to behave like a tutor.
Source: Simon Willison, co-creator of the Django web framework, managed to extract the system prompt of ChatGPT’s study together feature.
In the end, it also isn’t fair to conclude that JSON is entirely of no value. OpenAI also says that JSON is well understood by the model in contexts related to coding. Kirk Kaiser, a developer and the author of Make Art With Python, told AIM that JSON is helping one add more details inside a prompt where they otherwise wouldn’t — it may indeed be helpful in some cases.
Similarly, Nikunj Khotari, a partner at FPV Ventures, echoed the sentiment on X, saying, “Free form prompts just don’t work as well as people want. Adding structure forcefully through JSON gives them a modicum of success that they never achieved with free form.”
That said, it is also important for users to experiment and run tests in certain situations.
“The thing to keep in mind is that this is all new, too, so you might actually discover something better, for your specific workload for a specific model,” says Kaiser.
“But really, this points to why ‘engineering’ with LLMs is so challenging. They are impressive machines with unpredictable outputs,” he added.
The post Quit JSON, It’s Not Worth It for Prompting appeared first on Analytics India Magazine.
