Decoding MJPEG From USB Camera (ESP32-P4)

[besterm]

I have an ESP32-P4 Function Board that I'm using to decode MJPEG data from a USB Logitech C270 camera. I have the UVC demo working and can see MJPEG images at about 25FPS over WiFi to a PC. I am trying to add code to display the video on the LCD that is connected to the eval board. When I attempt to use the esp_new_jpeg component (https://components.espressif.com/compon ... p_new_jpeg) I get the following debug output error messages:

I (20421) example: Streaming...
E (20791) jpeg.decoder: JPEG_RESTART_INTERVAL configured by the software is non-0 but the RST marker cannot be parsed by the decoder
E (20791) jpeg.decoder: the number of data units obtained after decoding a frame of image is different from the number of data units calculated based on the image resolution configured by the software
E (20811) dma2d: dma2d_force_end(968): transaction not in-flight
E (20811) MPEG_DECODER: Fail to decode frame ret -1


I manually parsed the image header information:
SOI: FF D8
APPO MARKER: FF E0
LENGTH: 00 21
ID: 41 56 49 31 00
JFIF VER: 01 01
DENSITY UNITS: 01
X DENSITY: 00 78
Y DENSITY: 00 78
XTHUMBNAIL: 00
YTHUMBNAIL: 00

00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00

DQT: FF DB
LENGTH: 00 43
QT INFO: 00 (8 BIT TABLE ID 0)

08 05 06 07 06 05 08 07
06 07 09 08 08 09 0C 14
0D 0C 0B 0B 0C 18 11 12
0E 14 1D 19 1E 1E 1C 19
1C 1F 20 24 2E 27 20 22
2B 22 1B 1C 28 34 28 2B
2F 31 33 34 33 1F 26 38
3C 38 32 3C 2E 32 33 31

DQT: FF DB
LENGTH: 00 43
QT INFO: 01 (8 BIT TABLE ID 1)

08 09 09 0C 0A 0C 17 0D
0D 0D 17 31 21 1C 21 31
31 31 31 31 31 31 31 31
31 31 31 31 31 31 31 31
31 31 31 31 31 31 31 31
31 31 31 31 31 31 31 31
31 31 31 31 31 31 31 31
31 31 31 31 31 31 31 31

DEFINE RESTART INTERVAL: FF DD
LENGTH: 00 04
RESTART INTERVAL 16 MCUS: 00 0A

APP0 MARKER: FF E0
LENGTH: 00 04
00 00 // not sure the purpose of this data

START OF FRAME: FF C0
LENGTH: 00 11
PRECISION: 08
IMAGE HEIGHT: 01 E0
IMAGE WIDTH: 02 80
NUMBER OF COMPONENTS: 03
COMPONENT DATA: 01 21 00 02 11 01 03 11 01

IMAGE DATA: FF DA
LENGTH OF SOS SEGMENT: 00 0C
NUMBER OF COMPONENTS: 03
COMPONENT INFO: 01 00 02 11 03 11
SPECTRAL SELECTION PARAMS: 00 3F 00

COMPRESSED IMAGE DATA:
F0 1A 28 00...

In the image data, I can see the RST markers from FFD0 to FFD7 being repeated throughout the image.
I feel that the data is good, as I send the exact same data to the WiFi server to send to the PC. PC video looks good.

Any ideas?

[nopnop2002]

This library is bundled as standard with the ESP-IDF ROM.

https://github.com/espressif/idf-extra- ... /esp_jpeg/

idf component is here.

https://components.espressif.com/compon ... ions/1.3.0

The official sample is here.

https://github.com/espressif/esp-idf/tr ... /lcd/tjpgd

[besterm]

I think I found the problem. Evidently Logitech doesn't send standard MJPEG formatted video. I found this reference online:

The Logitech C270 webcam utilizes a proprietary, embedded H.264 stream within a Motion-JPEG (MJPEG) format. While MJPEG is a standard format, the C270 adds H.264 data to the JPEG frames as APP attachments, essentially creating a stream within a stream. This means that the output is not standard MJPEG and may require specific software to decode correctly.

[nopnop2002]

Thanks for the info.

I've always wondered if Logitech's webcam products don't follow standard specifications.

[tore-espressif]

Hello guys,
We tested USB streaming and MJPEG decompression from Logitech C270.
The stream misses some non-compulsory markers, but the esp-new-jpeg cannot decode without these markers.
But the problems are already fixed in esp-idf driver https://github.com/espressif/esp-idf/tr ... river_jpeg.

Example usage here https://github.com/espressif/esp-idf/tr ... peg_decode