Linux Start Guide for ST BrightSense Image Sensors

Supported Devices

This guide covers all ST BrightSense portfolio products, including the following image sensors:

Compatible ST BrightSense Hardware Tools

STMicroelectronics provides turnkey hardware tools for immediate integration onto Linux platforms, enabling instant plug-and-play connection to embedded processing platforms. The Linux drivers supported by this guide are developed to operate specifically on these hardware tools.

Each ST BrightSense image sensor is available with two main hardware kit options, each containing the necessary board, optics, and a 22-pin FFC/FPC cable.

The S-Board is a hardware kit including a sensor board for a given image sensor, equipped with an M12 lens holder and a removable lens, allowing users to change lenses at any time.

The P-Board is a hardware kit including a board with a connector for evaluating any camera module provided by STMicroelectronics and its authorized partners. This kit enables the evaluation of image sensors built as turnkey camera modules, saving effort associated with lens selection and focus.

Finding Further Product Information

ST BrightSense documentation and free software tools, such as Linux drivers, are publicly available on st.com. While this guide focuses on Linux driver installation and prerequisites, further information on product specifications and features can be found in the dedicated product documentation. Each product webpage includes a "Documentation" section for access to data briefs, datasheets, user manuals, and application notes. The "Tools & Software" section lists compatible hardware and software tools and provides links to their respective webpages for more information and downloads.

About Linux Camera Stack

The Linux camera stack is a collection of open-source components that enable the interfacing and control of camera devices on Linux-based systems. It encompasses both kernel-level and user-space components, providing a comprehensive framework for different camera applications, from simple webcam utilities to complex computer vision systems.

Figure 4. Linux Camera Stack

The diagram illustrates the Linux camera stack, showing user space components like Libcamera, v4l2-ctl, yavta, and media-ctl, interacting with kernel space components such as V4L2 framework, V4L2 drivers, and device tree. Hardware components include the ST Image Sensor, SoC, CSI driver, and ISP driver.

Introduction to V4L2 Framework

Video for Linux 2 (V4L2) is the kernel API for handling video devices. It provides a collection of drivers and a standardized API for camera drivers, allowing applications to access and control various video capture devices, including USB cameras, embedded camera modules, and hardware IPs involved in image processing. V4L2 exposes device nodes (e.g., /dev/videoX, /dev/v4l-subdevX) to user space, enabling standard V4L2 applications to control sensors and capture frames. Many applications support V4L2, facilitating straightforward integration of image sensors:

• v4l2-ctl: A command-line tool for querying and configuring V4L2 devices.
• yavta: "Yet Another V4L2 Test Application", a command-line tool for configuring and capturing frames from V4L2 devices.
• VLC: A media player with V4L2 support for capturing and streaming video from V4L2-compatible devices.
• GStreamer: A multimedia framework with V4L2 plugins for capturing, processing, and streaming video.
• OpenCV: A computer vision library with V4L2 support for capturing and processing video streams, widely used for image and video processing, object detection, and machine vision applications.

Introduction to Libcamera

Complementing V4L2 is libcamera, a user-space library providing a unified interface for camera applications. Libcamera abstracts the complexity of handling the media pipeline, offering a consistent API for camera control and image processing, and enabling support for advanced ISP features in modern SoCs. As a newer addition to the Linux camera stack, libcamera is dynamic and evolving, focusing on modernizing the camera stack and addressing the complexities of modern camera hardware and use cases.

Hardware Setup

Follow the instructions from Raspberrypi.com. Ensure the flex cable is plugged in correctly and before applying power.

Image Description: The image shows two electronic boards. The left board has a camera module attached via a flex cable, with a red 'X' marking indicating incorrect insertion. The right board, identified as a Raspberry Pi, shows a correctly inserted flex cable, marked with a green checkmark. Both boards have various connectors and components labeled.

V4L2 Driver

This section describes how to build and install the V4L2 driver on a Linux target platform. The driver is built directly on the target, for the corresponding target architecture and against the current running Linux kernel headers. Raspberry Pi is used as an example, but the process is similar on other Linux platforms.

Libcamera

#!/usr/bin/python3

from PyQt5 import QtCore
from PyQt5.QtWidgets import (QApplication, QHBoxLayout, QLabel, QPushButton,
 QVBoxLayout, QWidget)
from picamera2 import Picamera2,Preview
from picamera2.previews.qt import QGlPicamera2
import io
import time

picam2 = Picamera2()

# picam2.post_callback = post_callback
picam2.configure(picam2.create_preview_configuration(main={"size": (1120,
1360)},raw={"format": "R8"}))
picam2.set_controls({"ExposureTime":1000,"AnalogueGain":1.0,"AeEnable":False,"FrameRate":60})

print (picam2.sensor_format)

app = QApplication([])

def on_button_clicked():
    button.setEnabled(False)
    cfg = picam2.create_still_configuration()
    picam2.switch_mode_and_capture_file(cfg, "test.jpg",
    signal_function=qpicamera2.signal_done)

def capture_done(job):
    picam2.wait(job)
    button.setEnabled(True)

qpicamera2 = QGlPicamera2(picam2, width=1120, height=1360, keep_ar=False)
button = QPushButton("Click to capture JPEG")
label = QLabel()
window = QWidget()

qpicamera2.donesignal.connect(capture_done)
button.clicked.connect(on_button_clicked)

label.setFixedWidth(400)
label.setAlignment(QtCore.Qt.AlignTop)

layout_h = QHBoxLayout()
layout_v = QVBoxLayout()

layout_v.addWidget(label)
layout_v.addWidget(button)

layout_h.addWidget(qpicamera2, 80)
layout_h.addLayout(layout_v, 20)

window.setWindowTitle("Qt Picamera2 App")
window.resize(1120, 1360)
window.setLayout(layout_h)

picam2.start()
app.exec_()

v4l2-ctl

v4l2-ctl is a command-line tool for querying, configuring, or streaming from a V4L2 device. It is available in most distributions and can be installed on Debian-like systems with:

Useful one-liners:

• List formats and controls:
  $ v4l2-ctl --list-formats-ext --list-ctrls --list-ctrls-menu
• Select format (resolution: 640x480, pixel format: GRBG Bayer 10Bit):
  $ v4l2-ctl --set-fmt-video width=640,height=480,pixelformat=BA10
• Configure V4L2 controls (manual mode, exposure 1800 lines):
  $ v4l2-ctl --set-ctrl auto_exposure=1
$ v4l2-ctl --set-ctrl exposure=1800
• Configure Vertical Blanking to change framerate (vblank is resolution dependent):
  $ v4l2-ctl --set-ctrl vertical_blanking=3056
• Capture 20 frames (saved to capture_20f.raw10):
  $ v4l2-ctl --stream-mmap --stream-count=20 --stream-to=capture_20f.raw10

yavta

yavta (Yet Another V4L2 Test Application) is a command-line tool supporting many V4L2 capabilities for interacting with V4L2 devices. It can be installed on Debian-like distributions with:

Useful one-liners:

• List formats and controls:
  $ yavta --list-controls --enum-formats /dev/video0
• Select format (resolution: 640x480, pixel format: GRBG Bayer 8Bit):
  $ yavta --size 640x480 --format SGRBG8 /dev/video0
• Configure V4L2 controls (manual mode, exposure 1800 lines):
  $ yavta --set-control '0x009a090 1' /dev/video0
$ yavta --set-control '0x00980911 1800' /dev/video0
• Configure Vertical Blanking to change framerate (vblank is resolution dependent):
  $ yavta --set-control '0x009e0901 3056' /dev/video0
• Capture (30 frames saved in 30 files):
  $ yavta --capture=30 --file=frame-%.raw8 /dev/video0

V4L2 Code Example

To utilize the V4L2 API, you can develop your own application. Here is an example for streaming with Python.

First, install the packages for Python 3:

Example Python Code:

from v4l2 import *
import cv2
import fcntl
import mmap
import numpy as np

vd = open('/dev/video0', 'rb+', buffering=0)
print(">> get device capabilities")
cp = v4l2_capability()
fctl.ioctl(vd, VIDIOC_QUERYCAP, cp)

print(">> device setup")
fmt = v412_format()
fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE
fmt.fmt.pix.pixelformat = V4L2_PIX_FMT_SGRBG8
fmt.fmt.pix.width = 300
fmt.fmt.pix.height = 300

fctl.ioctl(vd, VIDIOC_S_FMT, fmt) # set whatever default settings we got before
fctl.ioctl(vd, VIDIOC_G_FMT, fmt) # get current settings
print("width:", fmt.fmt.pix.width, "height", fmt.fmt.pix.height)

print(">> init mmap capture")
req = v412_requestbuffers()
req.type = V4L2_BUF_TYPE_VIDEO_CAPTURE
req.memory = V4L2_MEMORY_MMAP
req.count = 2
fctl.ioctl(vd, VIDIOC_REQBUFS, req)

buffers = []
req.count = 2
for ind in range (req.count):
    # setup a buffer
    buf = v412_buffer()
    # tell the driver that we want some buffers
    buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE
    buf.memory = V4L2_MEMORY_MMAP
    buf.index = ind
    fctl.ioctl(vd, VIDIOC_QUERYBUF, buf)
    mm = mmap.mmap(vd.fileno(), buf.length, mmap.MAP_SHARED, mmap.PROT_READ |
                   mmap.PROT_WRITE, offset=buf.m.offset)
    buffers.append(mm)

    # queue the buffer for capture
    fctl.ioctl(vd, VIDIOC_QBUF, buf)

print(">> Start streaming")
buf_type = v412_buf_type(V4L2_BUF_TYPE_VIDEO_CAPTURE)
fctl.ioctl(vd, VIDIOC_STREAMON, buf_type)

while True: # capture 50 frames
    buf = v412_buffer()
    buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE
    buf.memory = V4L2_MEMORY_MMAP
    fctl.ioctl(vd, VIDIOC_DQBUF, buf) # get image from the driver queue
    mm = buffers[buf.index]
    image_data = np.frombuffer(mm, dtype=np.uint8, count=fmt.fmt.pix.bytesperline * fmt.fmt.pix.height).reshape(fmt.fmt.pix.height, fmt.fmt.pix.bytesperline)
    cv2.imshow("test", image_data)
    k = cv2.waitKeyEx(1)
    if k == 27:
        exit(1)
    fctl.ioctl(vd, VIDIOC_QBUF, buf) # requeue the buffer

print(">> Stop streaming")
fctl.ioctl(vd, VIDIOC_STREAMOFF, buf_type)
vd.close()