Texas Instruments AM6x Developing Multiple Cameras

Espesifikasyon

• Non pwodwi: Fanmi aparèy AM6x
• Supported Camera Type: AM62A (With or without built-in ISP), AM62P (With Built-in ISP)
• Done Sòti Kamera: AM62A (Raw/YUV/RGB), AM62P (YUV/RGB)
• ISP HWA: AM62A (Wi), AM62P (Non)
• Aprantisaj Pwofon HWA: AM62A (Wi), AM62P (Non)
• Grafik 3D HWA: AM62A (Non), AM62P (Wi)

Introduction to Multiple-Camera Applications on AM6x:

• Kamera entegre yo jwe yon wòl enpòtan nan sistèm vizyon modèn yo.
• Utilizing multiple cameras in a system enhances capabilities and enables tasks not achievable with a single camera.

Applications Using Multiple Cameras:

• Siveyans Sekirite: Enhances surveillance coverage, object tracking, and recognition accuracy.
• Antoure View: Enables stereo vision for tasks like obstacle detection and object manipulation.
• Sistèm Anrejistrè Kabin ak Miwa Kamera: Bay yon pwoteksyon pwolonje epi elimine tach avèg yo.
• Imajri Medikal: Offers enhanced precision in surgical navigation and endoscopy.
• Dron ak Imaj Ayeryen: Capture high-resolution images from different angles for various applications.

Connecting Multiple CSI-2 Cameras to the SoC:
To connect multiple CSI-2 cameras to the SoC, follow the guidelines provided in the user manual. Ensure proper alignment and connection of each camera to the designated ports on the SoC.

Nòt aplikasyon
Devlope aplikasyon plizyè kamera sou AM6x

Jianzhong Xu, Qutaiba Saleh

REZIME
This report describes application development using multiple CSI-2 cameras on the AM6x family of devices. A reference design of object detection with deep learning on 4 cameras on the AM62A SoC is presented with performance analysis. General principles of the design apply to other SoCs with a CSI-2 interface, such as AM62x and AM62P.

Entwodiksyon

Kamera entegre yo jwe yon wòl enpòtan nan sistèm vizyon modèn yo. Itilizasyon plizyè kamera nan yon sistèm ogmante kapasite sistèm sa yo epi pèmèt kapasite ki pa posib ak yon sèl kamera. Anba la a gen kèk egzanp.amples of applications using multiple embedded cameras:

• Siveyans Sekirite: Plizyè kamera ki plase estratejikman bay yon pwoteksyon siveyans konplè. Yo pèmèt yon sistèm panoramik views, reduce blind spots, and enhance the accuracy of object tracking and recognition, improving overall security measures.
• Antoure View: Multiple cameras are used to create a stereo vision setup, enabling three-dimensional information and the estimation of depth. This is crucial for tasks such as obstacle detection in autonomous vehicles, precise object manipulation in robotics, and enhanced realism of augmented reality experiences.
• Cabin Recorder and Camera Mirror System: A car cabin recorder with multiple cameras can provide more coverage using a single processor. Similarly, a camera mirror system with two or more cameras can expand the driver’s field of view epi elimine kwen mouri yo sou tout kote yon machin.
• Medical Imaging: Multiple cameras can be used in medical imaging for tasks like surgical navigation, providing surgeons with multiple perspectives for enhanced precision. In endoscopy, multiple cameras enable a thorough examination of internal organs.
• Drones and Aerial Imaging: Drones often come equipped with multiple cameras to capture high-resolution images or videos from different angles. This is useful in applications like aerial photography, agriculture monitoring, and land surveying.
• With the advancement of microprocessors, multiple cameras can be integrated into a single System-on-Chip.
  (SoC) to provide compact and efficient solutions. The AM62Ax SoC, with high-performance video/vision processing and deep learning acceleration, is an ideal device for the above-mentioned use cases. Another AM6x device, the AM62P, is built for high-performance embedded 3D display applications. Equipped with 3D graphics acceleration, the AM62P can easily stitch together the images from multiple cameras and produce a high-resolution panoramic viewYo prezante karakteristik inovatè SoC AM62A/AM62P yo nan plizyè piblikasyon, tankou [4], [5], [6], elatriye. Nòt aplikasyon sa a p ap repete deskripsyon karakteristik sa yo, men pito l ap konsantre sou entegrasyon plizyè kamera CSI-2 nan aplikasyon vizyon entegre sou AM62A/AM62P.
• Tablo 1-1 montre prensipal diferans ki genyen ant AM62A ak AM62P an sa ki konsène tretman imaj.

Tablo 1-1. Diferans ant AM62A ak AM62P nan Tretman Imaj

Konekte plizyè kamera CSI-2 ak SoC la
Sousistèm Kamera a sou SoC AM6x la gen konpozan sa yo, jan yo montre nan Figi 2-1:

• MIPI D-PHY Receiver: receives video streams from external cameras, supporting up to 1.5 Gbps per data lane for 4 lanes.
• CSI-2 Receiver (RX): receives video streams from the D-PHY receiver and either directly sends the streams to the ISP or dumps the data to DDR memory. This module supports up to 16 virtual channels.
• SHIM: a DMA wrapper that enables sending the captured streams to memory over DMA. Multiple DMA contexts can be created by this wrapper, with each context corresponding to a virtual channel of the CSI-2 Receiver.

Multiple cameras can be supported on the AM6x through the use of virtual channels of CSI-2 RX, even though there is only one CSI-2 RX interface on the SoC. An external CSI-2 aggregating component is needed to combine multiple camera streams and send them to a single SoC. Two types of CSI-2 aggregating solutions can be used, described in the following sections.

CSI-2 Aggregator Using SerDes
Yon fason pou konbine plizyè kouran kamera se sèvi ak yon solisyon serializasyon ak deserializasyon (SerDes). Done CSI-2 ki soti nan chak kamera konvèti pa yon serializè epi transfere atravè yon kab. Deserializè a resevwa tout done serialize ki transfere soti nan kab yo (yon kab pou chak kamera), konvèti kouran yo tounen an done CSI-2, epi answit voye yon kouran CSI-2 antrelase nan yon sèl koòdone CSI-2 RX sou SoC a. Chak kouran kamera idantifye pa yon kanal vityèl inik. Solisyon agregasyon sa a ofri avantaj anplis pou pèmèt koneksyon long distans jiska 15m soti nan kamera yo rive nan SoC a.

Serializatè ak deseryalizè FPD-Link oswa V3-Link yo (SerDes), ki sipòte nan SDK AM6x Linux la, se teknoloji ki pi popilè pou kalite solisyon agregasyon CSI-2 sa a. Tou de deseryalizè FPD-Link ak V3-Link yo gen chanèl bak ki ka itilize pou voye siyal senkronizasyon ankadreman pou senkronize tout kamera yo, jan yo eksplike sa nan [7].
Figi 2-2 montre yon egzanpampfason pou itilize SerDes yo pou konekte plizyè kamera ak yon sèl SoC AM6x.

Yon ansyenample of this aggregating solution can be found in the Arducam V3Link Camera Solution Kit. This kit has a deserializer hub which aggregates 4 CSI-2 camera streams, as well as 4 pairs of V3link serializers and IMX219 cameras, including FAKRA coaxial cables and 22-pin FPC cables. The reference design discussed later is built on this kit.

CSI-2 Aggregator without Using SerDes
Kalite agregatè sa a ka konekte dirèkteman ak plizyè kamera MIPI CSI-2 epi rasanble done ki soti nan tout kamera yo nan yon sèl kouran pwodiksyon CSI-2.

Figi 2-3 montre yon egzanpample of such a system. This type of aggregating solution does not use any serializer/deserializer but is limited by the maximum distance of CSI-2 data transfer, which is up to 30cm. The AM6x Linux SDK does not support this type of CSI-2 aggregator

Aktive plizyè kamera nan lojisyèl

Camera Subsystem Software Architecture
Figi 3-1 montre yon dyagram blòk wo nivo nan lojisyèl sistèm kaptire kamera a nan AM62A/AM62P Linux SDK, ki koresponn ak sistèm HW nan Figi 2-2.

• Achitekti lojisyèl sa a pèmèt SoC a resevwa plizyè kouran kamera avèk itilizasyon SerDes, jan yo montre nan Figi 2-2. SerDes FPD-Link/V3-Link la bay chak kamera yon adrès I2C inik ak yon kanal vityèl. Yo ta dwe kreye yon sipèpozisyon pyebwa aparèy inik ak adrès I2C inik pou chak kamera. Chofè CSI-2 RX la rekonèt chak kamera lè l sèvi avèk nimewo kanal vityèl inik la epi li kreye yon kontèks DMA pou chak kouran kamera. Yo kreye yon nœud videyo pou chak kontèks DMA. Done ki soti nan chak kamera yo resevwa epi estoke lè l sèvi avèk DMA nan memwa a kòmsadwa. Aplikasyon espas itilizatè yo itilize nœud videyo ki koresponn ak chak kamera pou jwenn aksè nan done kamera yo. Pa egzanp.amples of using this software architecture are given in Chapter 4 – Reference Design.
• Nenpòt chofè detèktè espesifik ki konfòm ak kad travay V4L2 a ka konekte epi itilize nan achitekti sa a. Gade [8] konsènan kijan pou entegre yon nouvo chofè detèktè nan SDK Linux la.

Image Pipeline Software Architecture

• The AM6x Linux SDK provides the GStreamer (GST) framework, which can be used in the ser space to integrate the image processing components for various applications. The Hardware Accelerators (HWA) on the SoC, such as the Vision Pre-processing Accelerator (VPAC) or ISP, video encoder/decoder, and deep learning compute engine, are accessed through GST pluginsVPAC (ISP) la li menm gen plizyè blòk, tankou Vision Imaging Sub-System (VISS), Lens Distortion Correction (LDC), ak Multiscalar (MSC), chak koresponn ak yon plugin GST.
• Figure 3-2 shows the block diagram of a typical image pipeline from the camera to encoding or deep
  learning applications on AM62A. For more details about the end-to-end data flow, refer to the EdgeAI SDK documentation.

For AM62P, the image pipeline is simpler because there is no ISP on AM62P.

Avèk yon ne videyo ki kreye pou chak kamera, kanal imaj ki baze sou GStreamer la pèmèt tretman plizyè antre kamera (konekte atravè menm koòdone CSI-2 RX la) an menm tan. Yon konsepsyon referans ki itilize GStreamer pou aplikasyon milti-kamera bay nan pwochen chapit la.

Konsepsyon referans

Chapit sa a prezante yon konsepsyon referans pou egzekite aplikasyon plizyè kamera sou AM62A EVM, lè l sèvi avèk Twous Solisyon Kamera Arducam V3Link la pou konekte 4 kamera CSI-2 ak AM62A epi egzekite deteksyon objè pou tout 4 kamera yo.

Sipòte Kamera
The Arducam V3Link kit works with both FPD-Link/V3-Link-based cameras and Raspberry Pi-compatible CSI-2 cameras. The following cameras have been tested:

• D3 Engineering D3RCM-IMX390-953
• Leopard Imaging LI-OV2312-FPDLINKIII-110H
• IMX219 cameras in the Arducam V3Link Camera Solution Kit

Setting up Four IMX219 Cameras
Follow the instructions provided in the AM62A Starter Kit EVM Quick Start Guide to set up the SK-AM62A-LP EVM (AM62A SK) and ArduCam V3Link Camera Solution Quick Start Guide to connect the cameras to AM62A SK through the V3Link kit. Make sure the pins on the flex cables, cameras, V3Link board, and AM62A SK are all aligned properly.

Figure 4-1 shows the setup used for the reference design in this report. The main components in the setup include:

• 1X SK-AM62A-LP EVM board
• 1X Arducam V3Link d-ch adapter board
• FPC cable connecting Arducam V3Link to SK-AM62A
• 4X V3Link camera adapters (serializers)
• 4X RF coaxial cables to connect V3Link serializers to V3Link d-ch kit
• 4X IMX219 Cameras
• 4X CSI-2 22-pin cables to connect cameras to serializers
• Cables: HDMI cable, USB-C to power SK-AM62A-LP and 12V power sourced for V3Link d-ch kit)
• Other components not shown in Figure 4-1: micro-SD card, micro-USB cable to access SK-AM62A-LP, and Ethernet for streaming

Configuring Cameras and CSI-2 RX Interface
Set up the software according to the instructions provided in the Arducam V3Link Quick Start Guide. After running the camera setup script, setup-imx219.sh, the camera’s format, the CSI-2 RX interface format, and the routes from each camera to the corresponding video node will be configured properly. Four video nodes are created for the four IMX219 cameras. Command “v4l2-ctl –list-devices” displays all the V4L2 video devices, as shown below:

There are 6 video nodes and 1 media node under tiscsi2rx. Each video node corresponds to a DMA context allocated by the CSI2 RX driver. Out of the 6 video nodes, 4 are used for the 4 IMX219 cameras, as shown in the media pipe topology below:

Jan yo montre pi wo a, antite medya 30102000.ticsi2rx la gen 6 sous pad, men se sèlman 4 premye yo ki itilize, chak pou yon IMX219. Topoloji tiyo medya a kapab tou ilistre grafikman. Egzekite kòmand sa a pou jenere yon pwen. file:

Then run the command below on a Linux host PC to generate a PNG file:

Figi 4-2 la se yon imaj ki pwodui lè l sèvi avèk kòmandman ki bay pi wo yo. Ou ka jwenn konpozan ki nan achitekti lojisyèl Figi 3-1 la nan graf sa a.

Streaming from Four Cameras
Avèk tou de pyès ki nan konpitè ak lojisyèl yo byen enstale, aplikasyon plizyè kamera ka fonksyone nan espas itilizatè a. Pou AM62A, ISP a dwe byen ajiste pou pwodui bon kalite imaj. Gade Gid Ajisteman ISP AM6xA a pou konnen kijan pou fè ajisteman ISP. Seksyon sa yo prezante egzanp.ampdifizyon done kamera sou yon ekran, difizyon done kamera sou yon rezo, epi estoke done kamera yo sou files.

Streaming Camera Data to Display
Yon aplikasyon debaz nan sistèm milti-kamera sa a se pou difize videyo ki soti nan tout kamera yo sou yon ekran ki konekte ak menm SoC la. Sa ki anba la a se yon egzanp sou kanalizasyon GStreamer.ample nan difize kat IMX219 sou yon ekran (nimewo nœud videyo yo ak nimewo v4l-subdev nan pipeline nan ap pwobableman chanje de rdemaraj an rdemaraj).

Streaming Camera Data through Ethernet
Olye pou yo difize done kamera a sou yon ekran ki konekte ak menm SoC la, yo kapab difize tou atravè Ethernet la. Bò reseptè a kapab swa yon lòt processeur AM62A/AM62P oswa yon PC lame. Men yon egzanp.ampfason pou difize done kamera yo atravè Ethernet la (lè w ap itilize de kamera pou senplifye) (note plugin kodè a ki itilize nan tiyo a):

Sa ki anba la a se yon ansyenampfason pou resevwa done kamera a epi difize l sou yon ekran sou yon lòt processeur AM62A/AM62P:

Storing Camera Data to Files
Instead of streaming to a display or through a network, the camera data can be stored in local files. Tiyo ki anba a estoke done chak kamera nan yon file (lè l sèvi avèk de kamera kòm yon ansyenamppou senplifikasyon).

Multicamera Deep Learning Inference

AM62A ekipe ak yon akseleratè aprantisaj pwofon (C7x-MMA) ak jiska de TOPS, ki kapab egzekite divès kalite modèl aprantisaj pwofon pou klasifikasyon, deteksyon objè, segmentasyon semantik, ak plis ankò. Seksyon sa a montre kijan AM62A ka egzekite kat modèl aprantisaj pwofon an menm tan sou kat flux kamera diferan.

Seleksyon modèl
The TI’s EdgeAI-ModelZoo provides hundreds of state-of-the-art models, which are converted/exported from their original training frameworks to an anembedded-friendlyy format so that they can be offloaded to the C7x-MMA deep learning accelerator. The cloud-based Edge AI Studio Model Analyzer provides an easy-to-use “Model Selection” tool. It is dynamically updated to include all models supported in TI EdgeAI-ModelZoo. The tool requires no previous experience and provides an easy-to-use interface to enter the features required in the desired model.

The TFL-OD-2000-ssd-mobV1-coco-mlperf was selected for this multi-camera deep learning experiment. This multi-object detection model is developed in the TensorFlow framework with a 300×300 input resolution. Table 4-1 shows the important features of this model when trained on the cCOCO dataset with about 80 different classes.

Tablo 4-1. Karakteristik prensipal modèl TFL-OD-2000-ssd-mobV1-coco-mlperf la.

Pipeline Setup
Figure 4-3 shows the 4-camera deep learning GStreamer pipeline. TI provides a suite of GStreamer plugins ki pèmèt dechaje kèk nan pwosesis medya yo ak enferans aprantisaj pwofon an bay akseleratè pyès ki nan konpitè yo. Gen kèk ansyenamples sa yo plugins gen ladan yo tiovxisp, tiovxmultiscaler, tiovxmosaic, ak tidlinferer. Tiyo ki nan Figi 4-3 la gen ladan tout sa ki nesesè yo plugins for a multipath GStreamer pipeline for 4-camera inputs, each with media preprocess, deep learning inference, and postprocess. The duplicated plugins pou chak chemen kamera yo anpile nan graf la pou yon demonstrasyon pi fasil.
The available hardware resources are evenly distributed among the four camera paths. For instance, AM62A contains two image multiscalers: MSC0 and MSC1. The pipeline explicitly dedicates MSC0 to process camera 1 and camera 2 paths, while MSC1 is dedicated to camera 3 and camera 4.

The output of the four camera pipelines is scaled down and concatenated together using the tiovxmosaic plugin. The output is displayed on a single screen. Figure 4-4 shows the output of the four cameras with a deep learning model running object detection. Each pipeline (camera) is running at 30 FPS and a total of 120 FPS.

Apre sa, se script konplè pipeline lan pou ka itilizasyon aprantisaj pwofon miltikamera a ki montre nan Figi 4-3 la.

Analiz pèfòmans

The setup with four cameras using the V3Link board and the AM62A SK was tested in various application scenarios, including directly displaying on a screen, streaming over Ethernet (four UDP channels), recording to 4 separate files, and with deep learning inference. In each experiment, we monitored the frame rate and the utilization of CPU cores to explore the whole system’s capabilities.

Jan yo te montre deja nan Figi 4-4 la, tiyo aprantisaj pwofon an itilize plugin tiperfoverlay GStreamer la pou montre chaj nwayo CPU a kòm yon graf ba anba ekran an. Pa default, graf la mete ajou chak de segonn pou montre chaj yo kòm yon pousantaj itilizasyon.tage. In addition to the tiperfoverlay GStreamer plugin, the perf_stats tool is a second option to show core performance directly on the terminal with an option for saving to a file. This tool is more accurate compared to the tTiperfoverlayas the latter adds extra load on theARMm cores and the DDR to draw the graph and overlay it on the screen. The perf_stats tool is mainly used to collect hardware utilization results in all of the test cases shown in this document. Some of the important processing cores and accelerators studied in these tests include the main processors (four A53 Arm cores @ 1.25GHz), the deep learning accelerator (C7x-MMA @ 850MHz), the VPAC (ISP) with VISS and multiscalers (MSC0 and MSC1), and DDR operations.

Table 5-1 shows the performance and resource utilization when using AM62A with four cameras for three use cases, including streaming four cameras to a display, streaming over Ethernet, and recording to four separate files. Two tests are implemented in each use case: with the camera only and with deep learning inference. In addition, the first row in Table 5-1 shows hardware utilizations when only the operating system was running on AM62A without any user applications. This is used as a baseline to compare against when evaluating hardware utilizations of the other test cases. As shown in the table, the four cameras with deep learning and screen display operated at 30 FPS each ,with a total of 120 FPS for the four cameras. This high frame rate is achieved with only 86% of the deep learning accelerator (C7x-MMA) full capacity. In addition, it is important to note that the deep learning accelerator was clocked at 850MHz instead of 1000MHz in these experiments, which is about only 85% of its maximum performance.

Tablo 5-1. Pèfòmans (FPS) ak Itilizasyon Resous AM62A lè yo itilize li avèk 4 Kamera IMX219 pou Afichaj Ekran, Difizyon Ethernet, Anrejistreman sou Files, ak Pèfòmans Enferans Aprantisaj Pwofon

Rezime
This application report describes how to implement multi-camera applications on the AM6x family of devices. A reference design based on Arducam’s V3Link Camera Solution Kit and AM62A SK EVM is provided in the report, with several camera applications using four IMX219 cameras, such as streaming and object detection. Users are encouraged to acquire the V3Link Camera Solution Kit from Arducam and replicate these examples. The report also provides a detailed analysis of the performance of AM62A while using four cameras under various configurations, including displaying to a screen, streaming over Ethernet, and recording to files. It also showsAM62A’sA capability of performing deep learning inference on four separate camera streams in parallel. If there are any questions about running these examples, soumèt yon demann nan fowòm TI E2E a.

Referans

1. AM62A Starter Kit EVM Quick Start Guide
2. ArduCam V3Link Camera Solution Quick Start Guide
3. Edge AI SDK documentation for AM62A
4. Edge AI Smart Cameras Using Energy-Efficient AM62A Processor
5. Camera Mirror Systems on AM62A
6. Driver and Occupancy Monitoring Systems on AM62A
7. Quad Channel Camera Application for Surround View and CMS Camera Systems
8. AM62Ax Linux Academy on Enabling CIS-2 Sensor
9. Edge AI ModelZoo
10. Edge AI Studio
11. Perf_stats tool

TI Parts Referred in This Application Note:

• https://www.ti.com/product/AM62A7
• https://www.ti.com/product/AM62A7-Q1
• https://www.ti.com/product/AM62A3
• https://www.ti.com/product/AM62A3-Q1
• https://www.ti.com/product/AM62P
• https://www.ti.com/product/AM62P-Q1
• https://www.ti.com/product/DS90UB960-Q1
• https://www.ti.com/product/DS90UB953-Q1
• https://www.ti.com/product/TDES960
• https://www.ti.com/product/TSER953

AVI ENPÒTAN AK DISCLAIMER

TI FONI DONE TEKNIK AK FIABILITE (KANK FYÈ DOE), RESOUS DESIGN (ENKLI DESIGN REFERANS), APLIKASYON OSWA LÒT KONSÈY KONSEYEN, WEB ZOUTI, ENFÒMASYON SEKIRITE, AK LÒT RESOUS "KÒMAN YO" AK TOUT FÒT, EPI LI RELIZE TOUT GARANTI, EXPRESSE AK ENPLIKITE, KI ENKLI SAN LIMITATION NENPÒT GARANTI ENPLIKITE KOMPLASANITE, ADAPTE POU YON OBJEKTIF PARTICULIER OSWA PWOPRIYETE PWOPRIYETE. .

Resous sa yo fèt pou devlopè kalifye k ap desine ak pwodwi TI. Ou se sèl responsab pou

1. chwazi pwodwi TI ki apwopriye pou aplikasyon w lan,
2. desine, valide, ak teste aplikasyon w lan, epi
3. ensuring your application meets applicable standards, and any other safety, security, regulatory, or other requirements.

These resources are subject to change without notice. TI permits you to use these resources only for the development of an application that uses the TI products described in the resource. Other reproduction and display of these resources is prohibited. No license is granted to any other TI intellectual property right or to any third party intellectual property right. TI disclaims responsibility for, and you will fully indemnify TI and its representatives against, any claims, damages, costs, losses, and liabilities arising out of your use of these resources.

Yo bay pwodwi TI yo dapre Kondisyon Vant TI a oswa lòt kondisyon ki aplikab ki disponib swa sou ti.com oswa bay ansanm ak pwodui TI sa yo. Pwovizyon TI a nan resous sa yo pa elaji oswa otreman chanje garanti aplikab TI a oswa deklarasyon garanti pou pwodwi TI.

TI rejte nenpòt tèm adisyonèl oswa diferan ou te pwopoze epi li rejte.

AVI ENPÒTAN

• Adrès Postal: Texas Instruments, Post Office Box 655303, Dallas, Texas 75265
• Copyright © 2024, Texas Instruments Incorporated

Kesyon yo poze souvan

Dokiman / Resous

Texas Instruments AM6x ap devlope plizyè kamera [pdfGid Itilizatè AM62A, AM62P, AM6x Devlope plizyè kamera, AM6x, Devlope plizyè kamera, plizyè kamera, Kamera

Referans

• Manyèl itilizatè