https://5734647.fs1.hubspotusercontent-na1.net/hubfs/5734647/Ebooks/Scaling%20Factory%20Expertise%20-%20Arch.pdf
Solving problems has always been a challenge, but the significance lies in being able to solve specific and critical issues rather than being a basic problem solver.
"Key Findings Across the Four Brain Groups
• Human Group 1: Expert SMEs
Expert subject matter experts (SMEs) achieved high accuracy as expected, drawing on their highly specialized skills. They were able to identify root causes and write effective
guidance in about 10 minutes on average. Their clarity suffered slightly; all experts in this group had decent English skills, but some were English as a second language (ESL)
speakers, as is common in factories. When asked about their guidance, the experts also self-reported reluctance to write highly detailed, prescriptive instructions, noting that
other employees are expected to have enough background knowledge rather than requiring line-by-line directions.
• AI Group 1: ChatGPT with a Basic Prompt
ChatGPT performed directionally correctly but lacked precise guidance. It finished in nearly exactly 10 seconds each time, but did not produce a solution fully usable for solving
the problem.
• Human Group 2: Engineering PhDs with a Basic Prompt
Three engineers with PhDs, operating under the same basic prompt, performed directionally correctly but with slightly worse root cause identification than ChatGPT, as
assessed by SME experts. However, they provided slightly better next steps and guidance on what to do. Overall, they were rated similarly to ChatGPT in usefulness, achieving about half the score of the expert SMEs, but took over 10 times longer to complete the task.
• AI Group 2: ArchFX Expert System
(Claude Sonnet with Expert Thought Pattern and Structured Data)
ArchFX, using the Claude Sonnet model combined with expert-designed thought patterns and structured data from Arch APIs and dashboards, consistently delivered
over 9/10 quality scores, often surpassing human expert performance in root cause identification and guidance quality.
After the initial comparisons, each part of the matrix was run three times and then stress tested with 200 retries to check for variance or hallucination. Using Sonnet 3.7, the ArchFX
System achieved 100% accuracy. With the older Sonnet 3.5 model—where some critical information was available only via dashboard screenshots—optical character recognition
(OCR) errors occurred in 5 out of 200 cases. These minor OCR errors slightly affected the usability of the end response but did not cause broader hallucinations and were found
only in the older models, which are no longer used in production systems.
All models using the expert thought pattern wrote useful, appropriately detailed prescriptive guidance without fatigue, achieving results more than 10 times faster, and
at over 100 times less cost compared to a mid- to senior-level manufacturing engineer’s salary in the United States, based on time to solution"
