[We asked an AI what it thought the pitfalls of electronic product development were. This is the final article in the series provoked by the AI’s response.]
It has been an interesting exercise, converting an AI’s output into this series of articles. We have come to some conclusions about AIs and their uses, and how that reflects the general understanding of product development.
The most obvious thing is how limiting the rules-following nature of Large Language Models can be. The AI divided up the development process into sections (of which more later), and in each section listed three possible pitfalls. In misapplying the “rule of three” this way, it missed out an awful lot of pitfalls that we advise clients about. For the scoping phase of a project alone we have over a page full of potential issues we will want to look into.
The idea of the “rule of three” in writing is that you should give three examples to reinforce any point you want to make, because “What I tell you three times is true.” This principle of repetition dates back well before The Hunting Of The Snark, indeed “repetitio mater studiorum” (“repetition is the mother of learning”) was a Roman proverb. That is not what the AI did here. The potential pitfalls are not examples, and there is no repetition in the way they are presented. Score that as a fail for the AI.
The other striking point of the AI output is its brevity. We created five articles that were deliberately kept short (under 1000 words) from a document that was itself just as short. A different prompt might have generated a more loquacious document, but a straightforward prompt didn’t produce very much output considering the scope of the question. The definitiveness of the wording does not help here; while the document doesn’t say it outright, there is a strong implication to the reader that the issues presented are the only ones you need to consider. In such a short document that clearly misses so much, that does not inspire confidence.
Enough bashing the AI, though. Far more interesting are the conclusions we can draw from what it chose to present.
LLMs shape their choices from the data they are trained on, which has the same sort of problem with self-reinforcing bubbles as we see in social media. In our case, that means the structure of the output article itself is a consequence of that data, the opinions and attitudes that people have written down about electronic product development. That makes the slide from prototyping to preparing for production a particularly interesting choice.
We consider that there are at least three stages between the start of prototyping and the start of production, not two. Prototype development is followed by “real” product development, a very distinct stage, long before production is on the horizon. It looks like the prototyping process has been misunderstood when the prototype stage segues so quickly into production. That matches our experience of clients’ expectations, and it’s an issue with the public perception of prototypes that we’ve written about before.
A prototype is not your first attempt at making the product. It is the vehicle through which you find out how to make that first attempt, and preferably make it the last attempt you need. It happens early in the whole process, so that the information gained in creating it can inform the final design. We suspect this is not well understood, and many developers skip over it on the assumption that they know the answers already. It’s hard to overstate what a mistake that is. You may get lucky, but it’s far more likely that you will hit the issues that a prototype would have shaken out, and find them a lot later in the design process. You will also find them a lot harder to deal with.
We also thought that the trust issue brought up as one of the potential manufacturing pitfalls (that manufacturers may cut corners if they aren’t watched like a hawk) was both interesting and disappointing. It is rare in our experience; with a responsible manufacturer and appropriate production testing there is no benefit to cutting corners. Issues can arise when manufacturers have to substitute for components that are no longer available, but that’s a very different matter that we mention as part of potential supply chain issues. Still, the AI chose to include quality drift as something to highlight over the myriad other issues of quality control and production testing, the latter of which isn’t mentioned at all. Either it occurs or more likely people claim it occurs often enough to register with the AI, when the blindingly obvious step of production testing didn’t. We don’t think this sort of active distrust is helpful in building the relationships you need for a successful product.
What AI-generated articles have you seen recently, and what can you tell from them?