Developing Embedded Automotive Products

Developing Embedded Automotive Products Embedded products for automotive applications typically follow a very rigid development process; the details vary from Original Equipment Manufacturers (OEMs), but the need for risk mitigation is the same It's not unusual for processes, tools, and techniques to be similar within a specific industry, and the automotive industry is a case in point. Automotive development for both heavy vehicles and cars involves a set of development tools, techniques, and processes designed to provide a standardized approach to delivering a quality embedded product. Typical, too, of the automotive as well as many other industries are the various advocacy groups that promote this expected approach. One such group is the Automotive Industry Action Group, or AIAG (Chrysler Corp., Ford Motor Company, General Motors Corp.: 1995). The AIAG publishes an overview of processes, from concept through production phase, in its Advanced Product Quality Planning and Control Plan (APQP) Reference Manual. (See Table 1.) In general, the phases are: • • • • •

Conceptualization Product design and development Process design and development Product and process verification Launch.

The APQP approach is usually implemented as a high-level waterfall model. However, it offers no restrictions to spiral or agile approaches to software development. The manual specifies a minimum number of deliverable documents and activities. It does not specify a limit to the amount of documentation, although the formality of the Production Part Approval Process (PPAP) requires a submission - in full form - of at least 18 documents, often many more. The higher-level automotive quality standard is ISO/TS 16949:2002, which is a superset of the ISO 9001 quality system standard with an automotive spin. The 16949 standard does not explicitly prescribe APQP, but APQP aptly meets the requirements of the standard. Detailing the Approach There are many embedded parts on today's automobile, and managing all of the suppliers is difficult and time-consuming. Potential suppliers to automotive organizations are rigorously scrutinized as to business aspects as well as technical capabilities. The conceptual phase is usually started with a Request for Proposal (RFP) from a customer attempting to communicate his or her development needs through specifications, interviews, meetings, and reviews. Many organizations have a particular set of methods and criteria for this evaluation. Some approaches are proprietary; others use well-established methodologies. The end result captures the customer's needs or requirements and identifies any conflicting requirements. This sets the scope of the project - and failure in this phase translates to probable failure at the

end of the project. The voice of the customer surpasses the voice of the engineer, unless a safety concern is presented. APQP documentation suggests that the development team use techniques to benchmark the capabilities of competing companies. In the event of a novel or new product, this competitive benchmarking is unlikely. Benchmarking is often difficult, but it is possible to research product specifications in various trade journals and even to purchase and dismantle similar systems or subsystems from a competitor to determine performance capabilities and thus establish realistic targets. Once the need and performance requirements have been identified, a response to the RFP is formulated and issued. This makes the most use of the supplier's expertise and generates multiple concepts for the customer organization. Pre-approved suppliers submit their proposals to achieve the product objectives, largely addressing fit, key product interfaces, and end-customer interactions with the product. Their interpretations generally consist of drawings and physical models, though they can also make use of software simulations or a digital mockup (DMU). Proposals are often followed up with presentation materials, additional drawings, and a rough outline of cost and delivery possibilities. Eventually these activities produce a chosen supplier, and formal technical documentation of the product begins in earnest, including detailed hardware and software requirements. By the conclusion of the concept phase, the development team will have identified the design direction, the supplier, and the product goals, and enlisted management support for the project. This gives the development team a good foundation and a clear picture of the scope it is allowed for subsequent phases of product and process development. Product and Process Development Now that the direction for the product is clear, the development team can produce a set of specifications identifying the product's behavior in terms of software, hardware, and durability requirements. These specification documents describe not only the desired "normal" functional behavior, but also the environment in which the product will function, as well as failure behaviors. There are automotive standards that assist with these definitions (for example, the Society of Automotive Engineers standard SAE J1455). These documents go beyond providing development targets; they are also used to confirm that the design has achieved targets and met performance demands during the test and verification phase. Time to market is prized in APQP organizations, and they tend to develop products and processes simultaneously, or nearly simultaneously. Sometimes process design and development will start after product design and development commences, but otherwise they run in parallel. During the product development work, there are multiple releases of the product. These iterations have an increasing level of capability and functional maturity. Some organizations assign part numbers for each iteration of the hardware and software; any change to form, fit, or function for either portion requires new part numbers. Other organizations may treat the hardware and software as one part by assigning a new part number as if the aggregate were one entity.

To bridge any gap during these intermediate deliveries and the specifications, release notes are provided by the designing organization. These notes provide the customer organization and test and verification personnel with the known or perceived capability of the product. In the early going, engineers theorize how the design may behave; however, to confirm these theories, the designers may execute simulation runs, and engineering verification tests will be conducted whenever possible on the parts. Typical design confirmation activities include: • • • • • •

Calculations Simulation Engineering verification activities Functional and system integration testing Design Failure Modes and Effects Analysis Other design reviews.

Design reviews are part of the product development process for automotive applications. The reviews have a number of goals, some specific to the design (engineering peer reviews), some from the organizational perspective (manufacturing and service). Experience suggests that more frequent and diligent reviews will provide a more effective end result. Organizational policy will dictate the minimum set and type of reviews. The embedded team may opt for more reviews within the development group. The automotive world employs a very powerful tool in Design Failure Modes and Effects Analysis (DFMEA). The DFMEA requires a cross-disciplinary team of development engineers to analyze the functions of the product using a logical approach. One such example is the functional analysis system technique (FAST). This technique is used to develop a table that captures failure modes, causes, effects, and qualitative values related to these events. The tool works best when teams use it to solve design issues while working with the design, or in advance of the final design. It is possible to use FAST for embedded software development as well. The DFMEA is a design tool for upfront thought on potential problems. This allows the development team or project manager to focus the test cases upon the areas that present the most probable and/or highest impact, narrowing the focus of the design effort and verification efforts. Production processes also reflect the increasing sophistication and capabilities of the product under development, and they benefit from the same types of techniques and critiques as the development work. Production processes are proposed and verified and, where possible, recycled from existing production methods, such as: • • • • • •

Production process development and documentation Trial production runs Production process capability measurements and corrections Gauge repeatability and reproducibility Process Failure Modes and Effects Analysis Other process design reviews.

Like the DFMEA tool used for the design, the Process Failure Mode and Effects Analysis (PFMEA) tool critiques the production process, generating actions and changes in the

proposed processes or confirmation actions for the processes. Indeed, any business that requires processes can benefit from a critical review and identification of areas that pose a risk to success or customer satisfaction. This early critical review, followed by actions and follow-ups, improves the service even before the service is available. Product and Process Verification Any embedded software development team will also need to consider design verification. No matter the organization, an independent verification and validation (IV&V) team should be included. The objectivity of the verification team must not be tainted by the development efforts. After all, those who have developed the software and hardware have already made interpretations of the specifications and customer requirements. This independent group will have its own interpretations of the specifications, and will develop test specifications accordingly. Some organizations outsource parts of the verification activities. This is especially true where the environmental aspects of the product are concerned, because this equipment represents a high dollar investment. Verification activities are done in parallel with the development activities. They provide feedback to the product development and manufacturing process teams for improvement in their respective areas. Software functional and component performance verification is part of the delivery from the supplier. The results of these tests form part of the release notes for the individual software/hardware package. In the automotive world, a summary document called a Design Verification Plan and Report (DVP&R) is used to capture all of the activities that will verify that the design meets or exceeds requirements. This document provides a synopsis of the test conducted as well as the results of the testing. The activities in the DVP&R should be derived from the DFMEA, which has a detection column specifically for this purpose. A minimal design verification plan should accomplish the following: • • • • • •

Fully exercise all inputs (stress to limits) Exercise all output similarly Use random testing to provide unexpected input/output combinations Use as much of the system hardware as is available (other hardware on the data bus) Provide some level of extreme testing to push the software/hardware combination to the limit Confirm the requirements are met.

Acquiring material in the early phases is quite costly. Specially tooled prototype parts in small volumes take time from production lines, and standalone prototype lines that are only used occasionally carry a high price burden per piece. In the automotive world, this cost prohibition is overcome in part via simulation. The teams have access to a multitude of simulators that can simulate the hardware and software components of the product as well as any data bus. The same hardware in the loop simulator can often serve as a development fixture to simulate component and feature interactions with the new proposed component and feature set.

Other possibilities include products such as Matlab/Simulink®, LabView®, and similar tools capable of communicating on the various vehicle buses and analog-to-digital conversions. These tools are not just verification tools; they also facilitate creation of requirements documents or specifications that detail the component as well as functionality. In a well-equipped software team, in-circuit emulators (ICEs) are available for both development and early testing of the code. These allow hardware and software interactions to be observed in real time (actually, pseudo-real time) and provide early verification of the software in the target system. One of the defects of most ICEs is that they possess their own power supply, which sometimes will affect the verisimilitude of the emulation. Both emulation and simulation allow for product feature refinement in advance of the product's availability, as well as early verification before the prototype stage. An ICE provides a software/hardware replica of the normal processor (sometimes using the real processor), whereas a simulator does not have to provide the hardware level of support. Usually, an embedded developer will use simulators to provide data traffic for the bus. Manufacturing development is quasi-parallel with the product development phase. The intention is to shorten the delivery time for the product as well as to ensure that it can be manufactured. Additionally, the team will develop automated manufacturing testing of the product's software and hardware. In cases where sophisticated electronics are available, the software can test itself - for example, using a boundary scan. Software can also be written to help self-calibrate the product. In addition, software can be used to provide extreme conditions - for example, data bus overloading. The parts created from the rapidly intensifying production processes can be used by the customer for additional verification activities. These parts, level 2 and level 3, are used for component level and systems or vehicle integration verification activities. This level of verification is crucial because it provides real platforms for the embedded product, going beyond what can be achieved during the early stages using emulators and simulators. As the design and production processes mature (part level 4 and 5), the parts are used for increasingly demanding activities, such as vehicle durability confirmation, which often includes harsh weather and driving conditions. Vehicles equipped with these parts may spend significant time in severely cold and hot regions, accruing many miles during multi-state, multi-regional trips and providing critiques of everything from product performance to sounds that may emanate from the component. These tests represent the extremes of the product use. Process validation activities are designed to prove the capability of the production processes. Even a perfect product is useless if a part cannot be produced at the intended volumes. Validating processes requires a well-documented production process. All hardware and software, for the product as well as the manufacturing processes, are frozen. The production processes are then stressed via run-at-rate, which produces the product at the parts per minute proposed to meet the customer's needs. These parts are then validated to establish the "process capability" of the production line - in other words, an evaluation of the line ability. Launch

Verified product is put onto early-build vehicles, which are used for plant material handling as well as validation. Product documentation is developed for the manufacturing facility installing the product. This documentation is based upon the technical specifications and functional documents. Suppliers offer technical support during the early launch activities. The developed embedded software can be subject to a number of possibilities after launch and during manufacturing. One programming option is post-launch product programming, either during manufacturing or at the customer's site with special tools. Additionally, maintenance of and upgrades to the software must be considered when developing its architecture. Product traceability and configuration management demands extend past the initial production launch. Closing the Loop The feedback, evaluation, and corrective actions portion of the development is not really a phase, but rather activities or systems that run parallel to the project. As in most industries, rapid feedback of test results is a prerequisite to solving non-conformance and quality problems quickly. Key metrics are identified and monitored. Metrics that move outside the expected performance, indicating a trend toward non-conformance, require corrective actions. Sometimes these corrective actions extend past the product launch. Automotive product development organizations use a tool known as the eight disciplines, or "8D" model, whose steps are as follows: 1. Assemble a cross-functional team with the required expertise. 2. Define the problem. 3. Implement and verify interim containment actions - temporary fixes. 4. Identify and verify root cause. 5. Identify and verify permanent corrective actions; preventive actions are also chosen. 6. Implement and validate permanent corrective actions. 7. Prevent recurrence of the problem/root cause. 8. Recognize the efforts of the team. Emergency situations require emergency actions, such as stopping the manufacturing line immediately if, for example, the verification team or the software team determines the software has a safety issue. For addressing problems in which it is possible to determine "good" units from "bad" units, containment actions will be started. (It may be possible to ship the "good" units, provided there is confidence in the ability to discern, via test, between the two states.) Permanent corrective actions require determining "root cause," tracing the failure back to the causing

element. Just to be sure, any remedial action taken must be verifiable to confirm that it has corrected the problem. While 8D particularly follows production issues, it is possible for the development team to use this tool to formally address problems found during the development work. It is important to note that the ultimate goal is to eliminate the element causing the problem. Experience suggests that frequently the true culprit is not identified; often a symptom is treated, but not the root cause or causes. Feedback at this point may sound as if it is coming a bit late in the process. In reality, if appropriate feedback is to be provided, the metrics that mean something to the product and project success must be identified early - then monitored vigorously by either technical staff or project managers, depending upon the metric. This feedback system communicates identified risks and allows for subsequent mitigation actions. Incorporating the feedback loop identified in APQP with a comprehensive risk management plan goes a long way toward reducing the risk of the project and product. A comprehensive test or verification approach will also reduce product and project risk exposure. One such technique, used by the Department of Defense, is the Test and Evaluation Master Plan (TEMP), which provides a comprehensive and systematic approach to verification activities. In this approach, the customer and the developers agree ahead of time to deliver a sequence of software packages. Each new software package is a superset of the previous package. Each package is fully functional. Each package has verification activities performed. The verification covers development verification, operational or integration verification, and deployment verification or live exercises. This approach allows for quick traceability of discovered problems to a unique software delivery. Not All About Automotive Apps The production process for an embedded product can be defined as a set of individual, incremental steps. Each step typically has a set of processes associated with it. The process control plan is a perfect tool for such sequences. The steps have key metrics identified to determine capability, and recurring measurements are taken to assess the state of the individual process. However, the use of this tool need not be restricted to the production process. Development processes and software processes can benefit from the same critical eye - in other words, the PFMEA. For areas found to be at risk, additional controls can be devised and placed on the process. The key to realizing the biggest benefit from DFMEA and PFMEA reviews is to perform them early in their respective process areas. Early use reduces development waste; it is cheaper to make changes to the product and process concepts and documentation than it is to make changes to equipment and processes already in place. One of the biggest strengths of the APQP model has to be the articulation of an easily understandable development process that encompasses customer input through to production.

This document provides a framework, not explicit detail that would force particular solutions from the adopting organization. Suppliers and customers can easily see how their processes map together, since both sides likely incorporate some content originating from this document. This mapping is further facilitated with the introduction of a common language and understanding of terminology. No matter where you go in the automotive world, these principles are known and used to some degree.