Save preview lockup defect

Summary

Using the TI DM368 SoC H.264 encoder and JPEG encoder hardware accelerators simultaneous can cause the TI provided video codec library to deadlock, requiring a camera power cycle to recover. This defect can be avoided by disabling frame view while saving.

Symptoms of the defect can be found by examining the messages log file:

http://10.11.12.13/static/log/messages

Look for lines similar to the following (but your will have timestamp/filestamp info as well):

replay_progress, memory status 0x31, readout status 0x02, software frame count 0x07, hardware frame count 0x 07, replay not complete 1
save_progress, released_buffers: 0, save_frames: 354
save_progress, released_buffers: 0, save_frames: 354
ERROR: looks like one or more frames were dropped.  Forcing EOS

If you find lines similar to these lines, you have encountered the concurrency defect.

Work around

For software version 2.3.1 and newer, set the Video encoding setting to Custom. This will create high encode quality videos (same as the High setting) but with preview during save disabled. During save, a black frame will be displayed. You can verify the work around is active by checking the metadata file associated with a captured video. In the metadata file you will find:

Pipeline description: H.264 high encode quality with preview during save disabled

For older software versions, you can download this pipelines_custom.txt file, put it on the SD card, power on the camera, then set the Video encoding setting to Custom.

Details

We first uncovered the TI concurrency defect Sept 2, 2015. Based on information from the TI website, we updated several software packages. The symptoms changed, but the underlying defect still exists.

During normal operation, the defect did not occur. We could get the defect to occur in endurance testing if we left the system run long enough. The defect did start occurring for customers when the camera was deployed in automated capture environments, such as baseball games. We were able to avoid the defect by simply disabling the web browser view of the save progress.

We found the following description of the problem:

Can H.264 and Mpeg/Jpeg run concurrently as they use different accelerators? The version 1.00.00 of H.264 and Mpeg cannot run concurrently. H.264 encoder uses VICP buffer for intermediate storage of data, hence Mpeg4 cannot run concurrently with H.264. But this does not limit Mpeg4 and H.264 multi-channel use case. This can be indeed achieved by running them sequentially in a frame interleaved manner. Version 2.00.00 H.264 encoder will have option to not to use the VICP/MJCP buffers. With those codecs, one can run VICP codecs and H.264 concurrently. But since both of them use EDMA channels, one should be sure that the sum of EDMA channels used by them should not increase the required limit.

We tried all the suggested changes to fix the problem, but without success. Based on the symptoms, it appears there is a concurrency defect in the TI supplied software library. We are unable to get access to the source code of the library, so we can not fix the defect. We work around the defect by avoiding concurrent use of the two hardware video encoders.