The world is changing fast. The pace of technological advancement increases every year, and today has produced high-speed multi-core processors; power-efficient systems-on-chip; and open, small-footprint operating systems with support for virtualization and high-performance computing. Together, these present a new world of opportunities for the development of embedded products delivering applications for cloud computing, machine-to-machine communication, and a plethora of other applications to improve our way of life.
But these major leaps forward come at a price. The code base necessary to run such sophisticated products has grown exponentially, bringing embedded project complexities to epic proportions. The interconnected nature of today’s products—part of the Internet of Things—has introduced greater complexities of being part of a grid, inherent security vulnerabilities and ever-increasing risk to successful project development and deployment. Further complicating projects are cost and time pressures as companies working with reduced budgets seek to minimize development costs and compress time-to-market schedules.
In the rush to market, it’s tempting to “build first and ask questions later” and to treat testing and quality-assurance as an afterthought. Results of an embedded-industry survey published in January 2014 found that of nearly 900 respondents, 72 percent had “late-cycle quality surprises” because the product was “used in unanticipated ways,” and that “edge” conditions were not tested. The article, titled “Time to Rethink Software Testing for Embedded Devices,” reported that 58 percent of such late-term surprises led to delivery delays and advocated negative-testing as a standard part of every QA regimen. In the article, “Taming Software Complexity,” embedded veteran Jack Ganssle counsels that limiting the number of paths through a program is the key to caging the complexity shrew.
A common thread between these practices is to enable teams that are normally involved in later phases of a development project to “shift left” their contributions and begin work sooner. Such teams typically include software developers, system integrators, and test engineers, who traditionally began their work after systems were designed (or apps were built) when remedial costs are at their highest. The shift-left approach (see Figure 1) brings these teams to the table sooner with requirements analysts, designers, and other inception teams to identify possible issues while they’re still in the planning phases and when they’re least expensive to address.
Figure 1: Shift-left diagram
Embedded-industry pioneer Wind River Systems has embraced the shift-left concept and is using it to help transform embedded development. "We are encouraging this concept of doing things differently," said Bill Graham, a director of product marketing at Wind River. "The basic concept of shift left is moving the discovery and mitigation of risks—both technical and business risks—to a point early on in the life cycle." He said that many of the risks related to embedded development have to do with hardware and software integration, and getting software running at a system level. "Those kinds of things are like a big-bang integration that happens late in the product cycle."
Wind River, a wholly-owned subsidiary of Intel, helps to mitigate the risks of such late-term integration with tools such as Simics*, a system simulator that can provide an accurate execution environment long before the target hardware is available. "Our customers are often working on pre-release hardware, and sometimes the hardware is not available," said Graham. And even in cases where hardware might be available early, "You would be lucky to get one board from the vendor if they had it," he added. Simics gives the entire team—designers, developers, engineers, and testers—access to the target hardware. Simics lets them simulate that hardware and perform tasks that they typically had to wait to do later in the lifecycle. Having that sort of access for the entire team has been a big deal.
Simulators today are just a few clicks away. But only a decade ago, developers were happy for the opportunity to buy time on a chipmaker’s latest silicon. Intel in 2002 introduced its first Early Access Program, which gave OS and app builders a crack at the Intel® Xeon® processor against which to test their applications. Today, embedded developers typically have a far broader set of prototyping requirements that often includes the entire target board plus a specialized debugging or testing harness. And that's just to bring up the operating system. Having access to a reliable, accurate software simulator can eliminate that entire first phase of a project. "That’s what we are seeing with Simics,” said Graham. The Simics environment can also simplify the enormously complex process of porting to different operating systems and targeting the huge variety of products out there.
During the design phase, simulation can help teams decide on which components are the most appropriate for the embedded product being built, and which are most effective in terms of power and cost. For example, a tester might point out early in a project that wireless Ethernet might not be well suited for use in a particular medical device due to security concerns or interference with other life-critical products. Similarly, when required apps are known up front, engineers might suggest one particular SoC over another for its ability to process the expected workload.
Indeed, software simulation might help prevent a problem that some organizations often don't realize they have until very late in the development process. "Because people are typically looking for the lowest power envelope, they usually want parts with just enough performance," said Graham. "When you discover that your hardware platform isn’t powerful enough, that’s a huge problem because you will either change the hardware, which rarely happens, or hack the software and try to make it work."
Simics is intended for the entire team, and according to Graham, it's priced accordingly. But despite its clear advantages, Graham said that some organizations have trouble seeing the value. "The ROI is usually very high, but often we have trouble convincing people of that," he said, despite a good track record with Simics. "That's why we suggest they review the entire portfolio. We help customers get over the hump that often puts things too late in the schedule. And it fits well with the whole agile or iterative approach. Pockets of people are doing this the right way because that shift left is about minimizing risk, cost, and schedule."
The Security of Things
As the world turns everyday conveniences into interconnected automatons, product designers on every project must be ever-wary about security. Never has the issue been more important than with today's trend toward the Internet of Things. From furnace to fridge, disc player to dishwasher, modern products increasingly are web-connected, browser-controlled, and field-upgradeable. To varying degrees, the same is true of public utilities, transportation systems, medical devices, automobiles, factory-floor machines, building security, and many other important systems.
The obvious benefits of such power and control at friendly fingertips also opens doors to malware and its makers, whose untiring inventiveness continues to confound the experts and cajole the unsuspecting. What's more, according to renowned security technologist and author Bruce Schneier, embedded software once deployed often doesn't receive patches. "For example, one survey of common home routers found that the software components were four to five years older than the device," he wrote in a recent blog post. For devices running Linux*, the survey showed, it was at least four years old, and the Samba* file system, when present, was at least six.
Embedded systems can protect themselves in numerous ways. Industry pioneer Wind River Systems employs a combination of hardware and software to protect systems running VxWorks*, Wind River Linux and its other real-time operating systems. "We have a root of trust system, starting with Linux and VxWorks Secure Boot," said Bill Graham, a director of product marketing at Wind River. The chain continues with encrypted communication between virtualized partitions and between products and cloud-based applications and management, limiting exposure of untrusted applications.
"Adding security or increasing security for embedded devices has become important across the board," said Graham. "These are not new applications, but they're new ways of dealing with security for applications that are already out there."
Even systems that adequately protect themselves at launch can become vulnerable later on, Schneier said, and too often stay that way. "Even when a patch is possible, it's rarely applied," blogged Schneier, the author of 12 books and hundreds of articles and papers. "Users usually have to manually download and install relevant patches. Because users never get alerted about security updates, and don't have the expertise to manually administer these products, it doesn't happen. ISPs sometimes have the ability to remotely patch routers and modems, but this is also rare."
The answer to embedding strong security, according to Graham, is to combine hardware and software with training and knowledge, plus improved testing and coding standards. "It's a tricky thing because it introduces a fair amount of new work and risk into a program, and I’m not sure people appreciate that. And while such up-front work might not be ideal,” said Graham, “the important thing is making sure that those products are protected from any dangers transmitted through the Internet by at least a couple of layers of defense."
Wind River's portfolio includes real software that's designed for specific applications and works with Android*, Wind River Linux*, and its VxWorks* real-time operating system. "Once the application has been tested, we are also seeing customers adopt the pre-integrated solutions from us and use our backend services to solve some key problems," said Graham. "This lets them concentrate on developing the actual application instead of messing around with their own platform."
The embedded world is ripe for innovation, with potential target markets that are as broad as they are diverse. Specialized platforms exist that are designed specifically for many such markets. Wind River alone offers platforms for automotive systems, medical products, industrial control, networking, robotics, infotainment, residential gateways, and consumer applications. It also offers a general-purpose platform. "We have an open-sourced based solution and a bare-bones, bare-metal hypervisor as well," said Graham, adding that these are not proprietary concepts. "We are implementing and getting it up and running—in this case on Intel hardware—so people can leverage that kind of approach. Wind River's professional services include development support for aerospace and defense, automotive, industrial, medical, mobile, machine-to-machine communications, and networking.” The company’s development and lifecycle management tools service all of its major embedded operating systems plus embedded virtualization.
Bigger Software, Same Team
Even with the increasing sophistication of development and simulation tools—and purpose-built hardware platforms with pre-integrated software—projects continue to fail in large numbers. Why? "The more things change, the more they stay the same," said Chris Rommel, vice president of machine-to-machine and embedded technology at VDC Research, which studies the industry. "There are some slight improvements if you were to compare 2013 against 2009 or 2010, but that's because R&D organizations were slightly more strapped with resources at the time. But a very large percentage of projects are still missing their schedules."
"You have cars now with over 100 million lines of code, which leads to issues around content creation and storage, wired, wireless and WAN connectivity, middleware, security and identity management, media storage, persistence, graphical-interface management, and so on. There are a lot of issues that some organizations don't have experience with." – Chris Rommel, vice president of machine-to-machine and embedded technology, VDC Research
In fact, embedded projects are burdened with a failure rate that's consistently between 30 and 60 percent, depending on how they're measured. That's a result of several factors, according to Rommel. "First and foremost, I'd put it on the growth of software content," and the added complexities therein. This alone would not be an insurmountable problem if companies were taking on corresponding staff to help manage it. "If you were to look at growth projections that we have for lines of code, it's growing much faster than you could expect an organization to add headcount. Such staffing increases would ordinarily be expected with incremental improvements or adoption of new technology, just as bodies might be added with a ‘new life cycle’ management tool or a new testing solution on the QA side," he said. But that's not happening in these development outfits.
Embedded products—with their Internet connections and machine-to-machine communications—require more software today than they did 10 years ago. And they will need still more ten years from now. "You have cars now with over 100 million lines of code, which leads to issues around content creation and storage, wired, wireless and WAN connectivity, middleware, security and identity management, media storage, persistence, graphical-interface management, and so on,” said Rommel. "There are a lot of issues that some organizations don't have experience with."
Suddenly, century-old companies are finding that in order to compete, they must embed software and user interfaces into their refrigerators, washers, and dryers. "You had corporate cultures that were very much centered around mechanical engineering; R&D organizations were built on and around hardware," said Rommel. Now they're also expected to be software engineering organizations. "Yes, and they're concerned not just with connectivity, but also with implementing a good level of security," added Graham. "And I think that’s where the complexity and risk with the Internet of Things concept is. You can do security scans and find a lot of these products open and on the Internet even today," said Graham. "What's even scarier is that some of these are factory-floor products connected directly to the Internet." (See The Security of Things sidebar later in this article.)
The bottom line is that the Internet of Things is forcing companies to rethink their product-development processes or risk finding themselves on the scrapheap of history. "In essence, this is a fundamental piece of end-product differentiation," said Rommel. "So not only are these organizations being asked to develop software, but it's also more and more critical that they get it right." Unfortunately, according to Rommel, more often than not they're missing the mark in terms of quality. "The embedded work in progress just doesn't work," taking most post-deployment update scenarios off the table. "It depends on the product class. One of the big challenges the embedded market must resolve is how to maintain certifications for products that are updated in the field, and when points of configuration can change. But still, if you ship something that doesn't work or it breaks, there can be tremendous liability on the backend."
Add to that the problem of managing all those software updates and tracking the products still in need, something that Google's nascent Android operating system seems to have figured out. Perhaps that's among the reasons it's one of the fastest growing operating systems for embedded. "Beyond mobile and tablets, if you look at where Android is being used, it's also in more of those once-upon-a-time headless systems," said VDC's Rommel, who specializes in machine-to-machine issues. "Now just about any embedded device with a screen or terminal has Android as an option in lieu of a total rework of those systems. It's enjoying high double-digit growth in medical devices, military communication products, and infotainment systems for cars. And that's in addition to the displacement of some certain device classes by repurposed, ruggedized kinds of consumer-grade tablets and such."
All in the Process
In the end, a product is only as good as the software it's running. If Google Glass* does little more than sit on your face and show maps, or if it turns out that evil-doers really can hack into someone's pacemaker, then the Internet of Things might be a just another flash in the pan. Getting an embedded product right the first time—on time—is a matter of getting everyone involved early in the process. "There has been a tremendous adoption of iterative and agile development methodologies in embedded in the past few years," said Rommel, and part of that has been a matter of survival. "You've had this organic growth in software-driven feature sets and existing applications," he said. "But then in the case of the car, it's also a conscious recognition and investment in an area that drives value recognition to the end customer. Ford has put out research on what causes people to buy the Focus*, for example. And it's very much because people really like the software experience in that vehicle."
Wind River's Graham agrees. "We have customers that have done—and are doing—incredible, innovative things. So what are they doing differently? They're getting smarter about doing things earlier in the lifecycle. Part of that is adopting agile, and part of that is using new technology to test beforehand so they're not doing it at the last minute." For organizations struggling with limited resources, tight deadlines, and shrinking budgets, Graham offers the assurance that, “There is a better way.”