I’ve taken part in designing and improving system architectures of various types, including CPU instruction sets. My skills range from high-level analysis of key issues such as overall throughput, implications of designs on cache hit rates, and so on, all the way down to low-level details such as instruction design. My experience includes processors from the original 4-bit microchip (an Intel 4004, I believe, though I was 15 years old when I first programmed some math functions for that chip for a friend) all the way through 128-bit VLIW (similar to EPIC) processors.
Often, while learning a new system, I quickly identify common mistakes in its design. Sometimes these mistakes are not compromises, though they might have been perceived to be so by the designers; they are mistakes that lead to lower performance or even persistent bugs in the implementations.
Having an experienced “outsider”, such as myself, review the architecture or design of your system, API, ABI, or critical-path software, is crucial to spotting problems early on. The costs of not doing so might remain hidden from financial analysis, but they can nevertheless pose a substantial long-term burden.
(In my experience, it is often much easier to spot problems in the works of others than in one’s own! Thoughtful, incisive questions regarding my own code, coming from relative outsiders, have led to some of my best “learning experiences”.)