We have already written an article on why prototyping is a good thing and you should plan to do it. Amusingly our AI prompter failed to identify not prototyping at all as a risk, but it made up for it by pulling out some interesting issues.
The first issue it raised was the temptation to rush. Getting your prototype hardware built quickly is obviously a good thing, as you can begin testing it sooner. Equally obviously, you want it to work. You shouldn’t skip your usual design checks and reviews just because what you are creating is a prototype.
Don’t go overboard, though. This is a prototype, and it isn’t expected to work perfectly. Its raison d’être is to find all the little faults that won’t otherwise occur to you. You want it to work well enough that you aren’t fighting your way through bugs that could have been caught before the board was built, but you do want it available in a timely manner without being held up for a fortnight in reviews.
The same goes for prototype software. Its purpose is to “prove” (test) the basic structure of the code and hardware functionality. By its very nature it won’t be perfect. It will be missing a great deal of the required functionality of your product, and any user interface will be simplistic. Those aren’t things that matter for a prototype. What does matter, and what you should review for, is that it does check out all basic functions of the design and examine all the risks and assumptions that you have identified so far.
That leads on to the second observation our AI prompter made. Prototypes may appear to work perfectly well in the lab, but real world flaws can go undetected for quite some time. There is a strong temptation when working with prototypes to do the same things time and again. Pressing buttons in the same order, sending the unit the same example data, or always using the same command script, all of these things can become problems. You want that sort of repeatability when you are chasing down a bug, but you need variation to trigger a bug in the first place.
Unit tests and test plans are your first line of defence here. Writing unit tests gives you an assurance that whatever changes you make to the code don’t break known working functionality. Test plans give you an assurance that you will exercise everything important, and break you out of the habits that lead you to only test one code path. As ever there is a balance to be struck. While prototyping, an excessively detailed test plan will bog you down for no later benefit. As you move towards final software and hardware, you will want to test more and more.
An important point here is to use real world data in your tests as much as is practical. We do make use of AI to generate test data, which has advantages in helping to tailor tests to deal with specific issues, but real world data will throw up odd cases that you may not think of in advance. If you have engineers who excel at this sort of testing, try to involve them even early in the prototyping phase.
The last point the AI raised was about user feedback. It is absolutely true that early users and beta testers will come across issues that developers won’t encounter for a variety of reasons. You need systems in place to capture that feedback, otherwise it will be lost and bugs will go unchecked. If you only have a few early users, this can be as simple as maintaining a spreadsheet, but a proper bug tracker such as Bugzilla is usually better.
However in raising this issue, we think the AI has itself fallen into a pitfall it failed to notice. It has blurred the line between prototypes and near production ready hardware and software to the point where it is easy to forget that there is a difference. Your prototypes are not the same as your final production units and will not behave the same way. Many of the points raised above apply much more usefully once you have moved past the prototyping phase and are beginning to develop on final hardware. Design reviews for production PCBs are more necessary and should be more careful. Unit tests for prototype software may be meaningless to production software. Bugs found on prototype units may not be reproducible on production hardware.
You really must take pains to regard your prototypes as just that; prototypes. Otherwise you are at risk of trying to sell your prototype, which will lead to the world of problems we described in another article.