Renesas R-Car V3H, V3M Guidebook

Introduction

This guidebook introduces the features of Renesas Electronics' R-Car V3H and V3M. It serves as reference material for customers investigating before adoption or considering applications. Renesas Electronics aims to expand possibilities for new product creation by consolidating information relevant to R-Car for customers. We will continue to enhance the content based on customer feedback.

Note: Content may be updated or changed without notice.

Table of Contents

• 1. Features and Applications of R-Car V3H, V3M
• 2. Other Functions of R-Car V3H, V3M
• 3. Camera Connection to MIPI-CSI2
• 4. Camera and ISP
• 5. How to Use R-Car's ISP
• 6. What is IMR (Image Rendering Unit) Distortion Correction Unit?
• 7. What is IMP (Image Recognition Engine)?
• 8. CNN and Toolchain
• 9. Status of Various OS Support
• 10. Renesas Electronics R-Car V3H, V3M Provided Software
• 11. Starter Kit Introduction
• 12. Evaluation Board Introduction
• 13. Evaluation Software Package
• 14. Functional Safety Package
• 15. Security Package
• 16. R-Car Consortium Partner Introduction
• 17. R-Car SIer and Utilization

Notes on the Guidebook

• This guidebook introduces application examples of R-Car but does not guarantee their content. Verification of feasibility according to the customer's intended use and application is necessary. Application examples may be changed without notice.
• For R-Car specifications, please consult the hardware manual or contact us via "Online Consultation".
• The content described in this guidebook is subject to change or deletion without notice.

1. Features and Applications of R-Car V3H, V3M

R-Car V3H, V3M are equipped with hardware features for Computer Vision, and this section introduces examples of using these functions in applications.

Figure 1 is a block diagram abstracting the functions of R-Car V3H, V3M. Key features include ISP, IMR, IMP, and CNN, and their functions and applications are introduced.

1. ISP (Image Signal Processing)

This function processes RAW data output from CMOS sensors, allowing custom image processing in combination with the sensor. If using an ISP-equipped camera module, the image from the camera module can be used as is. R-Car's ISP is used when custom image processing is required for applications like smart cameras, 3D surround view, and driver monitoring.

2. IMR (Image Rendering Unit) Distortion Correction Unit

This unit is used for correcting lens distortion and perspective transformation. This function allows for faster processing compared to CPU processing, reducing CPU load for distortion correction and perspective transformation.

3. IMP (Image Recognition Engine)

This is an accelerator for high-speed image recognition processing, improving efficiency through hardware and software. This accelerator can process various necessary tasks for image recognition at high speed.

4. CNN

This is a module that enables high-speed execution of Convolutional Neural Networks. Using the R-Car CNN Toolchain, it can be used in combination with various deep learning open-source libraries.

Figure 1: Abstracted Function Block Diagram

The diagram shows the R-Car V3H/V3M system with key functional blocks. Input is received via MIPI-CSI2. The system includes ISP, IMR, IMP, and CNN modules. These modules are connected to VIN (Video Input), VSPD (Video Signal Processor), DU (Display Unit), and LVDS (Low Voltage Differential Signal) outputs.

2. Other Functions of R-Car V3H, V3M

1. VIN (Video Input Module)

The video input module provides color space conversion functionality. If connecting a camera with ISP, input is from VIN.

2. VSPD (Video Signal Processor)

This module handles image overlay, rotation, and scaling. It is used for image processing in applications like surround view.

3. DU (Display Unit)

This unit receives image data from VSPD and outputs it via LDVS. It adjusts display size, position, format, and plane overlay.

4. LVDS (Low Voltage Differential Signal)

This unit receives RGB signals (display control and data) from DU, converts them to LVDS, and outputs various timing signals to drive LCD panels.

5. MIPI-CSI2 (Mobile Industry Processor Interface/Camera Serial Interface2)

This is a receiver module for Camera Serial Interface2, supporting MIPI CSI-2 V1.1 and D-PHY V1.1.

3. Camera Connection to MIPI-CSI2

Camera input is necessary for front cameras, surround monitoring, driver monitoring, etc. R-Car V3H, V3M can connect cameras via MIPI-CSI2 (Figure 2 connection example). Figure 2 shows direct connection, but to extend connection distance, a serializer/deserializer is used (Figure 3). A serializer converts parallel signals to serial for high-speed serial communication with a deserializer, which outputs parallel signals. This is used to extend camera/image signal connection distances.

Figure 2: Connecting a camera to MIPI-CSI2

The diagram shows a simplified R-Car V3H/V3M system connected to a camera via MIPI-CSI2. The camera block includes ISP and a serializer. The R-Car block includes ISP, IMR, IMP, CNN, VIN, VSPD, DU, and LVDS.

Figure 3: Camera connection using serializer/deserializer

This diagram illustrates a camera connection to R-Car V3H/V3M using a serializer and deserializer. The camera, equipped with ISP and a serializer, connects via MIPI-CSI2 to the R-Car, which has ISP, IMR, IMP, CNN, VIN, VSPD, DU, and LVDS. The data flow is shown as YUV data.

MIPI-CSI2 has 4 input lanes, with specifications differing between R-Car V3H and V3M:

• R-Car V3H: Supports 6Gbps input for a single camera. For four cameras, it supports 1.5Gbps input per camera.
• R-Car V3M: Supports 4Gbps input for a single camera. For four cameras, it supports 1.0Gbps input per camera.

Figure 4 shows the connection of four cameras.

Figure 4: Connecting four cameras using serializer/deserializer

This diagram depicts the connection of four cameras to the R-Car V3H/V3M system using serializers and deserializers. Each camera's RGB data is processed through its ISP and serializer before being sent via MIPI-CSI2 to the R-Car, which includes ISP, IMR, IMP, CNN, VIN, VSPD, DU, and LVDS modules.

4. Camera and ISP

Cameras can be equipped with or without an ISP. Figure 5 shows a camera with ISP, and Figure 6 shows a camera without ISP. Image sensors output RAW data, which needs ISP processing for image viewing or recognition due to format differences. Cameras with ISP offer pre-prepared processing, but customization is difficult and increases cost. Using R-Car's ISP can reduce camera costs. For image recognition, especially with monochrome cameras or custom processing, R-Car V3H, V3M's ISP is suitable.

Figure 5: Camera with ISP

The diagram shows a camera with an image sensor that outputs RAW data. This RAW data is then processed by an ISP (Image Signal Processor) which outputs YUV/RGB data. A serializer is connected to the ISP output.

Figure 6: Camera without ISP

The diagram shows a camera with an image sensor that outputs RAW data. This RAW data is directly sent to a serializer, bypassing an ISP.

Connection methods for cameras without ISP via MIPI-CSI2 are the same as for cameras with ISP. Figure 7 shows a single camera connection, and Figure 8 shows a four-camera connection.

Figure 7: Connecting a camera without ISP to MIPI-CSI2

This diagram shows a single camera without an ISP connecting to the R-Car V3H/V3M system. The camera's RAW data is sent through a serializer via MIPI-CSI2 to the R-Car, which includes ISP, IMR, IMP, CNN, VIN, VSPD, DU, and LVDS.

Figure 8: Connecting four cameras without ISP to MIPI-CSI2

This diagram illustrates the connection of four cameras, each without an ISP, to the R-Car V3H/V3M system. Each camera's RAW data is serialized and sent via MIPI-CSI2 to the R-Car, which processes it through its internal ISP, IMR, IMP, CNN, VIN, VSPD, DU, and LVDS modules.

5. How to Use R-Car's ISP

R-Car's ISP allows for image customization based on application needs, but requires dedicated software. Figure 9 shows the connection method for cameras without ISP and the ISP software configuration. The red-framed area in Figure 9 represents ISP software, which customers can develop themselves or customize based on ISP software provided by R-Car Consortium partners. This customization development is necessary.

Figure 9: Camera without ISP and ISP Software

The diagram shows a camera connected via MIPI-CSI2 and I2C to the R-Car system. The camera includes a sensor and serializer. The R-Car system features camera application software, custom application software (including sensor abstraction, auto white balance, auto exposure control, noise canceller, calibration, tuning), sensor driver, ISP firmware, ISP low-level firmware, I2C driver, OS, and an ARM Cortex-A53 processor. The ISP software is highlighted within a red box.

Note for customers considering R-Car's ISP:

Support is provided from ISP consulting to implementation. Proposals are made according to customer use cases and applications. Please consult us.

6. What is IMR (Image Rendering Unit) Distortion Correction Unit?

IMR is dedicated hardware for correcting image distortion. Development requires software.

Fish-eye lens distortion correction (3D surround view)

Fish-eye lenses capture a wide view but distort images, making subject identification difficult. IMR corrects this distortion. It can also be used to create overhead views by synthesizing images from multiple cameras.

Camera lens distortion correction

Camera lenses can cause images to appear non-flat or distorted. IMR's distortion correction function flattens and corrects these distortions. This dedicated hardware accelerator reduces CPU load.

Figure 10: Camera Lens Distortion Correction

The diagram illustrates the process of correcting lens distortion. An initial image with distortion (e.g., from a fish-eye lens) is shown, followed by the corrected image, which appears flat and undistorted.

Figure 11: Reduction of CPU Load by IMR

This diagram compares the CPU load for image processing. In the first scenario, processing is done solely by the CPU. In the second scenario, the load is split between the CPU and IMR, showing a significant reduction in CPU load when IMR is utilized for distortion correction.

Note for customers using IMR:

Consulting and development support for IMR usage and software development are provided. Please utilize this service.

7. What is IMP (Image Recognition Engine)?

R-Car V3H, V3M have an accelerator optimized for image recognition. Using this accelerator requires dedicated libraries written in accelerator-specific instructions, provided by Renesas Electronics. This accelerator's API differs from OpenCV's highly abstracted API, requiring changes to the API portion of OpenCV programs. Figure 12 shows the difference between OpenCV API and IMP library API.

Figure 12: IMP Image Recognition Engine and OpenCV

The diagram illustrates the relationship between image recognition applications and the underlying software/hardware. An "Image Recognition Application" can use either an "IMP Library API" or an "OpenCV API". The IMP path involves "Accelerator Command", "IMP Library", "Driver", and "Runtime Framework" (IMP-X5+), leading to the IMP Image Recognition Engine. The OpenCV path leads to "OpenCV API" and requires "Change" to interface with the IMP accelerator. The diagram highlights that the IMP library API is distinct from the OpenCV API.

Note for customers considering IMP library:

Training on IMP library specifications and usage, and support for API changes from OpenCV are available. Customers considering evaluation and product development with the IMP library are invited to consult.

8. CNN and Toolchain

Deep learning is used for tasks like object detection and recognition from camera input. Deep learning models are trained by configuring layers (e.g., convolutional operations) in a network structure to achieve high recognition rates.

The CNN Toolchain converts pre-trained deep learning network models into a format executable on R-Car V3H, V3M. It selects the optimal accelerator (IMP, CNN, etc.) based on the model's layers and converts it to an executable format for the device.

Details can be downloaded from the following Renesas Electronics site: Renesas R-Car CNN Toolchain

Figure 13: Toolchain and Deployment to R-Car

The diagram shows the workflow from deep learning model development to execution on R-Car. It starts with a "Deep Learning Framework" and a "Trained Network Model". This model is processed by the "Renesas CNN Toolchain", which outputs a "Command LIST" containing instructions for accelerators like "IMP-Core" and "CV engine". This is then used by the "Image Recognition Application" running on the "R-Car V3H", utilizing the "IMP Library" and "Runtime Framework". The final execution involves the "IMP Image Recognition Engine" and "CNN".

Note for customers considering CNN:

Consulting is provided for image recognition, deep learning, and tool usage in conjunction with IMP. Training and introduction support are available.

9. Conclusion

Production: Hitachi Industry & Control Solutions, Ltd.

As an R-Car SIer (System Integrator), Hitachi Industry & Control Solutions, Ltd. provides services such as sub-license sales of R-Car software products and platform construction with various OSes.

• [In-vehicle SoC Development Consulting Services]: https://info.hitachi-ics.co.jp/product/in-vehicle_sol/
• [R-Car Online Consultation]: https://info.hitachi-ics.co.jp/product/in-vehicle_sol/contact/
• [Contact]: kumikomi-hiics@ml.hitachi-ics.co.jp

Production/Editing/Reporting: Mitsuru Suwa (Hitachi Industry & Control Solutions, Ltd.) - Edited this guidebook based on experience and knowledge gained through customer-facing roles, including FAE activities and consulting for IVI, meters, ADAS, and other automotive equipment applications.

Production Members:

• Chukyu Haruhisa (Hitachi Industry & Control Solutions, Ltd.) - R-Car SI contact, responsible for customer guidance and consulting.
• Kazuhito Kawaguchi (Hitachi Industry & Control Solutions, Ltd.) - Involved in developing image recognition and deep learning using IMR, IMP, CNN for R-Car V3H, V3M. Responsible for introduction guides, OS support, and mass production design consulting.
• Tsutomu Nakajima (Hitachi Industry & Control Solutions, Ltd.) - Involved in developing camera connections and ISP image processing for R-Car V3H, V3M. Responsible for customer guidance, consulting, and mass production design.

Supervision/Cooperation:

• Goshi Momiyama (Renesas Electronics Corporation) - Responsible for R-Car V3H, V3M product marketing.
• Yasuhiro Yamakoshi (Renesas Electronics Corporation) - Responsible for marketing and business development for ADAS-focused R-Car.