360° in one view

Mr Bertsch, how many years has Liebherr-Components been involved with digital camera technology? How has this knowledge impacted the development of your LiXplore® Bird's Eye 360° surround view system?

Alexander Bertsch: We have been producing digital Ethernet cameras for mobile machines for more than ten years, today in the third generation. Our in-depth expertise in this field is no coincidence. It is based on the long-standing partnership with our customers and the insights we have been able to gain as a result.

A major plus for our customers is certainly that we know the demanding conditions under which their mobile machines are used - large temperature differences, vibrations, changing light conditions, etc. very well. We use this knowledge to develop our products according to the demands placed on them in terms of robustness. Accordingly, the experience we have gained over the last few years has helped us a great deal in developing the Bird's Eye 360° surround view system. Basically, it is a digital camera-monitor system that provides operators with a reliable all-round view of the working environment of their machines. The intelligent algorithm behind the display controller creates the 360° view by combining the individual images from four digital cameras into a complete picture.

Difficult operating conditions in the field, such as dust, moisture or vibration, as well as a partially restricted field of vision, do not exactly make it easy for the operator. This starts with classic construction machines, such as wheel loaders, and stretches up to agricultural machines, e.g. during harvesting. Municipal and special-purpose vehicles, such as waste disposal vehicles also face this problem.

In order to depict blind spots and thus avoid personal injury and damage to property, perfect visibility is indispensable. The sharper the image, the better, because the easier and more pleasant work becomes is for the operators. In concrete terms, operators who spend eight hours a day in their cabs have a limited attention span - just like the rest of us. On top of that, they have to deal with the complexities of operating the machine. It is obvious to use a system that makes work easier for humans: a win-win situation for everyone involved, the operators and the machine manufacturers. Fewer errors equal lower costs and greater efficiency.

Many machine manufacturers have been using cameras to monitor processes and increase efficiency for some time. What is the benefit of digital systems compared to analogue? Couldn't multiple mirrors be used instead?

Alexander Bertsch: The answer is yes, it's possible, but it's no fun (laughs). You can think of it like a tube TV compared to a full HD LED TV with a 40 inch diagonal. A digital camera has more power, a sharper image, richer contrasts and less distortion. In other words, it is an assistance system that actually gives operators greater convenience. And not only that: If you think about the future in terms of automation, connectivity and the autonomisation of machines, digital cameras do not only open up completely different functionalities, but are also a prerequisite for them. So it makes perfect sense to invest in the field with foresight. We would like to turn the "someday" into a "now", in order to remain competitive as a machine manufacturing company. However, investing in a digital surround view system is not entirely cost-effective.

Wouldn't it suffice to use a conventional surveillance camera and adapt it to the respective application?

Alexander Bertsch: Over the last few years, we have observed the following: Our customers know the challenging conditions in which their machines are used, and thus also the demands placed on the assistance systems. These include demanding environmental conditions in the field as well as changing light conditions, for example when working underground. However, all these conditions must be taken into account, when choosing a suitable visual aid. Our experience shows that customers often underestimate these requirements or, for cost reasons, use cameras that are unsuitable for the extreme conditions mentioned.

I believe that it is important to consider the total investment. If you include all the important factors in the decision, you save money and time as well as valuable nerves. Let's look at a simple calculation example: A construction machinery manufacturer buys four ordinary surveillance cameras for one of his excavators. The purchase costs are only 200 euros - a temptingly cheap investment. However, after the installation and test use of the visual aids, it quickly becomes clear that the cameras must be equipped with an additional protective housing, in order to further function reliably. This means that the costs for the housings and the installation are added to the originally quite low purchase costs. Thus, in the end, one is at double the originally calculated amount.

Another problem is that customers often do not consider the overall process behind the use of an assistance system. Before commissioning, such systems must first be calibrated, which is often time-consuming and cost-intensive due to the complex geometry of mobile machines. Unfortunately, this subsequent adjustment is often forgotten, but it causes non-negligible additional costs.

We see that the purchase of a "standard camera", such as a surveillance camera, is thus unfortunately a naive assessment of the situation. You can also compare this with a personal example. If one tries to save money when building a house by doing the work oneself, despite a lack of knowledge, instead of hiring a craftsperson, one saves money at first. However, if you "botch the job" because you don't know what you're doing, it will come back to haunt you, when you move in. The correction of this mistake ultimately costs more than it would have cost, if a craftsperson had been commissioned from the beginning.

It has also been shown that the calibration process in particular poses major challenges for many customers. If a machine manufacturer equips five of his machines with a surround view system, the calibration effort is still limited. However, if we are talking about several hundred machines, things look very different.

Therefore, we wanted to develop a time-saving system that can be calibrated within a few minutes. In concrete terms, this means that neither the calibration mats have to be placed exactly nor the distances to other objects have to be measured.

In addition to the 360° view, our LiXplore® Bird's Eye allows customers to define detailed views and customisable overlays according to their requirements and to assign the corresponding function keys to the display. This is even more convenient for the operators, because they can quickly switch between the views at the touch of a button.

How can you summarise the essence of your assistance systems?

Alexander Bertsch: The features of our assistance systems are the result of the extensive experience we have gained over the years with mobile machines in a wide range of industries. Our product is therefore "off the shelf", so to speak, but is, nevertheless, excellently suited for use in demanding environments. Due to our constant proximity to customers, we know the requirements for the components very well. We use this expertise to further develop our products and continue offering the greatest possible benefits.

What does the future hold for camera-monitor systems and surround view solutions?

Alexander Bertsch: As in most other areas of life, we are moving towards a completely digital future. Digital image processing will soon find its way into even simple camera-monitor systems.

Instead of several individual components that have to be linked, you get a complete solution with an additional assistance function that is already state-of-the-art. In the course of developing Bird's Eye, we kept asking ourselves the following questions: How can we achieve the greatest possible benefit for our customers and at the same time differentiate ourselves from other providers? Which technologies that have proven successful in other industries have enough potential to be adapted to the requirements of mobile machines? One example is collision warning, which is already being used in the automotive industry. Soon it will also find its way into the area of mobile machinery.

Today's systems still have to cover these extended assistance functions via additional sensors. With digital technology, intelligent algorithms take over this task. It is a leaner system overall that reduces possible sources of error and takes comprehensive account of processes on the construction site. Moreover, there is the additional convenience for the operators. An interface that displays all essential functions in one is far easier and more intuitive to operate.

Analogue cameras will continue to exist, but I see them more in simpler applications, such as material handling. For more complex requirements and larger machines, like harvesters, digital solutions will be essential.

If we think about artificial intelligence and a networked, autonomously working construction site that has to process large amounts of data, digital cameras are a mandatory prerequisite. The way towards this goal is not far, the most important steps have already been taken - both in research and in practice.

Through joint projects with universities and research institutes, we create synergies between the two areas. And we are also applying the knowledge we have already gained in Lindau from the development of artificially intelligent systems to digital image processing.

However, there are still a few hurdles on the road ahead of us until we get there. Who is responsible, if an autonomously driving machine causes personal injury or damage to property?

Alexander Bertsch: In my opinion, this almost ethical issue is still an open book and a work in progress from a political standpoint. Assistance systems must, therefore, be developed in the future in such a way, that they are able to compensate for malfunctions in automation and minimise risks in business operations. Striking a balance in this area of tension between artificial intelligence and functional safety will, in my view, still be a giant task.