Why AI Gets Industrial Product Questions Wrong | ReshapeX

[Skip to main content](#main-content)[Skip to navigation](#navigation)[Skip to footer](#footer)

June 29, 2026

Juan Aparicio

# Why AI Gets Industrial Product Questions Wrong, and What Actually Fixes It

[← Back to blog](/en/insights)

I gave a talk at Automate last week on the shift from chatbots to agents in industrial sales and service. This is the argument I made, written down for the people who asked.

It starts with a question I hear in almost every customer conversation: when are the hallucinations going to be fixed? why if I just wait for the next model? when will I be able to use ChatGPT to reliably answer the daily questions I get from my customers? The assumption behind it is that hallucination is a defect, and that some future model will patch it out. Hallucination isn’t a defect waiting on a patch, and once you see why, the whole problem looks different

A language model doesn’t look things up. It predicts the next most likely token from patterns in its training data. So when you ask it to cross-reference a discontinued NOSHOK pressure transmitter, it gives you a fluent, confident answer, and it may hand you a PTI15 equivalent with PNP output when the customer specified NPN. Those aren’t interchangeable. The sensor powers up, reads correctly on its own display, and never trips the PLC input card. Nothing looks wrong for days. When it finally surfaces it looks like a dead sensor rather than a bad recommendation, and a tech loses two days chasing a fault that was never there. Add the downtime and the return and that one wrong crossover costs ten to seventy-five thousand dollars.

![Slide: "Hallucinations don’t show up in demos. They show up on the dock." The NOSHOK pressure transmitter crossover — wrong output ordered, signal never lands, customer escalates, true cost $10K to $75K+.](/images/blog/why-ai-gets-industrial-product-questions-wrong-inline-1.png)

The model wasn’t malfunctioning. It returned the most statistically probable answer, which is a different thing from the correct one. I put four real Siemens CPUs on a slide, all genuine products, with the model’s confidence spread across them. It picks the highest-probability part. That part doesn’t fit the customer’s existing rack, and the model has no way to know that, because the rack was never in its training data. This is the part to sit with: a bigger model doesn’t fix it. It picks the wrong part more confidently, which makes the mistake harder to catch.

![Slide: "Hallucinations are a feature, not a bug." Four real Siemens CPUs with the model’s statistical confidence spread across them; the 89% highest-probability match is the wrong physical fit.](/images/blog/why-ai-gets-industrial-product-questions-wrong-inline-2.png)

If a better model isn’t the fix, software is the place to look, because coding agents have started to genuinely work. Coding agents have a feedback loop. The compiler and the test suite are automatic verifiers. The agent writes code, the build runs, the tests pass or fail, and the agent reads the failure and tries again, usually before a human sees any of it. Software is unusual in that it ships with a free, instant, objective judge of whether the work is right.

Industrial product data has no such judge. There is no compiler for a part recommendation. No test goes red when you spec the wrong enclosure. The answer ships, and the only verification is a pallet arriving at a dock days later with the wrong thing on it, by which point nobody connects it back to the recommendation. You cannot build an automatic feedback loop here, because the thing that makes the software loop work, an instant verifier, does not exist in this domain and can’t be manufactured.

![Slide: "Why coding agents work, and industrial agents don’t." The software compiler loop (write code, run tests, read error, iterate) versus the industrial dock loop, which has no compiler and a 72-hour delay gap.](/images/blog/why-ai-gets-industrial-product-questions-wrong-inline-3.png)

What you can do is put people in the loop. Domain experts run the agent against real cases, find what it gets wrong, and the system gets tuned case by case. It is slow, and there’s no version of it that isn’t slow. You grind from the rough accuracy a general approach gives you up toward the 99.9 percent that industrial work actually requires, one corrected case at a time. The compounding payoff is that the same grind captures the knowledge those experts carry in their heads, the cross-references and exceptions that were never written down anywhere. The tuning produces a working agent. It also produces an asset.

That leaves the real question: what are you tuning? Three ideas describe what you can actually adjust, and they arrived in roughly this order. An open-book exam is the cleanest way to see the difference between them.

![The open-book exam framing: prompt engineering, context engineering, and the harness.](/images/blog/why-ai-gets-industrial-product-questions-wrong-inline-4.png)

Prompt engineering came first, around the end of 2022 when ChatGPT launched. It’s the wording of the question. A sharper question gets a sharper answer, but if the fact isn’t in the student’s head, no rephrasing puts it there. Prompting shapes tone, format, and how the model approaches a task. It can’t supply knowledge the model never had.

Context engineering is the open-book part of the exam. Now the student can bring material and consult it before answering. Andrej Karpathy gave the idea its name in a post on June 25, 2025, defining it as "the delicate art and science of filling the context window with just the right information for the next step." What you place in front of the model at answer time, the specs, the prior conversation, the relevant documents, matters more than how you phrase the question.

![The control panel retrofit example: a compressed conversation summary that drops the enclosure depth constraint.](/images/blog/why-ai-gets-industrial-product-questions-wrong-inline-5.png)

An open book introduces its own failure. Bring the wrong binder, or grab a page that looks right and isn’t, and you answer confidently and wrongly. When a conversation runs long, the agent compresses its own history to stay inside the context window. In one case I walked through, the customer had spent several messages describing a control panel retrofit: a 480V supply, a 24VDC control circuit, and tight enclosure depth. By the time the conversation turned to selecting contactors, the summary the agent had kept said only "needs 40A contactors." It recommended a part that matched on current and poles and carried a 120V coil, which won’t pull in on a 24VDC circuit, and ignored the depth constraint entirely. The agent didn’t lie. It couldn’t see what it had thrown away. A bigger context window doesn’t fix this. It moves the point where it breaks.

RAG, short for Retrieval-Augmented Generation, is also a popular way to expand what the agent knows. Before the model answers, a search step pulls relevant documents from your own data and places them in front of the model alongside the question, so the answer is built from your material rather than whatever the model absorbed in training. Connect it to the catalog and the datasheets and the model has the facts in front of it when it answers.

![The Rittal cabinet example: five real SKUs differing by a single digit, sitting almost on top of each other in vector space.](/images/blog/why-ai-gets-industrial-product-questions-wrong-inline-6.png)

It helps, but it has a ceiling, because of how the retrieval works. Most RAG systems retrieve by similarity: text gets converted into vectors, and the search returns the passages whose vectors sit closest to the query. That’s good for finding things by meaning and bad at telling near-identical things apart. Industrial catalogs are full of part numbers that differ by a single character and mean completely different products, a different output type, a different coil voltage, a different enclosure rating. To an embedding trained on meaning, those near-identical strings sit almost on top of each other, so the search returns the closest match rather than the correct one. The embedding doesn’t read a part number the way an applications engineer does; a suffix that changes the product reads as noise. A near-miss that ships the wrong part on a multi-position panel build isn’t a passing grade, it’s a recall.

The harness is the third idea, and the exam analogy carries it the rest of the way. The harness is like automating the structure of the exam itself to avoid the student to make any failure: which references the student may open, how they’re organized, what the student is allowed to write, and the rules that govern all of it. An agent is a model plus a harness. The model supplies intelligence and nothing else. The harness is everything around it, the tools the agent can call, the context it’s given, what it keeps between steps, and the limits on what it’s allowed to do. Prompting and context are two of the things a harness manages. Every coding agent that works is a harness built around a model. The model is the part everyone has. The harness is where reliability is engineered.

It’s worth being precise about what the harness does and doesn’t do, because the easy mistake is to imagine it as a fact-checker that knows right from wrong. It isn’t. The harness doesn’t verify answers. What it does is give the agent disciplined, structured access to the right tools and the grounded data those tools reach, and hold it to working from that rather than generating freely. For the use cases that ReshapeX tackles, the most important thing those tools reach is the knowledge graph: the products, their attributes, and the relationships between them, compatibility, replacements, required accessories, lifecycle status, modeled explicitly rather than left as loose text. That structure is what lets the agent answer a question by traversing real connections between real parts instead of guessing which datasheet sounds closest. Retrieval finds documents that look similar. The graph encodes how things actually relate. The harness gives the agent access to that graph, along with live data like inventory and pricing, and constrains it to build answers from there. The harness raises the ceiling on how accurate the agent can get. The human testing loop is what reaches that ceiling, case by case, until the answers hold.

![Left: knowledge graph of one of the biggest industrial automation companies in the world. On the right, close view of an orchid.](/images/blog/why-ai-gets-industrial-product-questions-wrong-inline-7.png)

Left: knowledge graph of one of the biggest industrial automation companies in the world. On the right, close view of an Orchid.

One more piece determines whether any of this is practical, which is that the harness has to be reusable, not rebuilt for every channel. The approach we’ve taken at ReshapeX exposes the harness as an MCP server. MCP (Model Context Protocol) is the open standard Anthropic introduced in late 2024 that the major labs and companies have since adopted. In plain terms, the same grounded layer serves the same answers whether the customer arrives through web chat, an ERP, a voice line, or a sales team’s Teams channel.

![Slide: "Three things to take back to your team on Monday" — stop asking which model, demand a grounding layer, build for portability.](/images/blog/why-ai-gets-industrial-product-questions-wrong-inline-8.png)

If there’s one thing to take back to your team, it’s a change in the question you ask vendors. Not which model do you use, because everyone has the same models and that’s not where the difference lives. Ask what every answer is grounded against. Ask how the system gets tested, and by whom, and how it gets from a convincing demo to something reliable in production. Ask what happens when it’s wrong. Those questions separate an agent from a chatbot with good manners, and the difference only shows up after you’ve deployed it.

## Give us your twenty hardest questions.

We’ll demo on your SKUs, run your evals, and cite every answer.

*   Real Examples
*   Working Demo
*   Your Data

Schedule a meeting Talk to the agent first