Captured video queue control

From edgertronic high speed video camera
Jump to navigation Jump to search

Background

Internal to the camera is a queue that holds the captured, but not saved, videos. When you trigger the camera, a video is captured and added to the queue. The camera will either automatically save the video (when save mode is set to auto, background-fifo, or background-lifo) or the camera will let you review the video and save the section you care about (save mode set to review-before-save).

A new manual save mode has been added to allow an external device to have direct control over the queue that holds captured videos. Captured videos are still added to the queue in the same way - every trigger adds a new captured video. The difference is in manual save mode the controlling external device has to specify which captured video to save using the CAMAPI selective_save() method. The controlling external device uses CAMAPI delete_captured_videos() method to remove captured videos from the capture queue when they are no longer needed.

The manual save mode is similar to review-before-save save mode with these key differences:

  • There is no webUI support for manual save mode. You have to program an external controlling device to use manual save mode.
  • In manual save mode, the save is in the background, so the camera can continue to be triggered to capture new videos while the camera is saving a previously captured video.
  • In review-before-save save mode and manual save mode, the entire captured video queue is emptied by calling CAMAPI run() method.
  • In manual save mode, you can selectively free up room in DDR3 for more captures by using CAMAPI delete_captured_videos() method.

Client control over camera's captured video queue

When the camera is triggered, a video is captured to the DDR3 memory and added to the captured video queue. Once the DDR3 memory is full of captured videos, triggering the camera is disabled. The captured videos remain in the queue (and thus in DDR3 memory) until the captured video is discarded (or the camera loses power).

When the camera's save mode is configured for auto, background FIFO, or background LIFO, the camera is in control, automatically saving and discarding captured videos from the queue. Camera control of the captured video queue means once the camera settings are configured, the user simply needs to trigger the camera, and the camera takes care of the rest.

The camera also supports a review before save configuration where the user via the web user interface, or a client application, controls the encoding parameters, starting and ending frames to save, and the order the captured videos are saved. When configured for review before save, there is no support for background save and all the captured videos are discarded when the CAMPAPI run() method is invoked thus making the camera ready to capture more videos.

The new manual save mode uses background save so new videos can be captured while a previously captured video is being saved. Manual save mode means a user-supplied computer application controls the camera's captured video queue. Manual save mode is not available via the camera's web user interface.

Queue control

The camera supports the manual save mode allowing a software client application control over how the captured videos are processed. A captured video can be:

  • Saved, using CAMAPI selective_save() method.
  • Saved, specifying the starting and end frame to save, using CAMAPI selective_save() method.
  • Saved, first changing some of the encoding parameters using the CAMAPI configure_save() method.
  • Saved multiple times, changing encoding parameters and/or starting and ending frames to save.
  • Deleted, using the new CAMAPI delete_captured_videos() method.
  • Deleted, including deleting all videos in the capture queue by calling the CAMAPI run() method.

Unfortunate terminology

To maintain compatibility with existing software that controls an edgertronic camera, some dictionary key names are used that no longer reflect their actual meaning. Had a more insightful engineer anticipated all the cool things you can do with a high-speed camera, they would have picked more generic names in the first place.

  • get_captured_video_info() key: unsaved_count: the actual meaning is the number of videos in the captured video queue.
  • get_captured_video_info() key: first_buffer: has no meaning when save mode is manual. Instead you have to walk the keys in the get_captured_video_info() returned dictionary and check for the value of the key having a dictionary data type. Sorry about that. The key will also be an integer (but the python JSON library forces it to be a string).
  • In filename parameter expansion, &b is referred to as the multishot buffer: the actual meaning is the capture number.
  • selective_save() key: buffer_number: the actual meaning is the capture number.

Identifying queued videos

The CAMAPI get_captured_video_info() method provide information about all the captured video in the queue. The get_captured_video_info() response has been extended to include two new dictionary keys:

  • user_parm - a string that can be provided with the CAMAPI trigger() method.
  • target_filename - the expanded base filename, without any directory paths. The actual filename could be different if specified via the CAMAPI selective_save() method.

The buffer_number is used with the CAMAPI selective_save() and delete_captured_videos() methods. Note that buffer_number is used for historic reasons with capture ID number being a more descriptive name. When CAMAPI run() is called the buffer_number is set to 0 and incremented by one each time the camera is triggered (meaning the first captured video after run() menthod is invoked will have an capture ID / buffer_nummber of 1).

Manual save interaction with other CAMAPI methods

When the camera is configured for manual save mode, there are interactions with varous CAMAPI methods the developer should keep in mind.

CAMAPI Method Parameter Interaction
run() all All videos in the video capture queue are deleted. buffer_number is reset to 1 for the next capture. Must be called when the camera is not currently saving a video.
run() key: filename_pattern May effect get_captured_video_info() key target_filename. See filenaming section for details.
run() key: save_mode Must be set to SAVE_MODE_MANUAL.
trigger() key: base_filename May effect get_captured_video_info() key target_filename. See filenaming section for details.
trigger() key: user_parm Returned by get_captured_video_info() as part of captured video information.
selective_save() key: buffer_number Used to identify the video in the captured video queue to save. The buffer_namer is the same as returned by get_captured_video_info().
selective_save() keys: start_frame
end_frame
Allows a subset of the captured frames for a specific video in the captured video queue to be saved.
selective_save() key: filename Highest priority filename pattern. Overrides default filename and any filename set via the CAMAPI methods run() reconfigure_run(), or trigger().
save_stop() all Not supported in manual save mode.
get_captured_video_info() all The buffer number is used as a dictionary key to index the captured video on intered. The keys user_parm and target_filename enable associating a specific trigger() invocation with a resulting captured video in the queue.
delete_captured_videos() key: delete_list Deletes videos from the captured video queue using the buffer index. The buffer indices can be extracted from the dictionary returned by get_captured_video_info().
delete_captured_videos() key: delete_all Deletes all the videos from the captured video queue.
get_camstatus() key: unsaved_frame_count Only applies to the current video being saved, not to the other unsaved videos in the queue.

Live view

By using manual save instead of background save, you are able to see live view in between video saves. This can be useful in cases like capturing baseball pitches, when there always seems to be captured videos, with the camera catching up at the half inning.

Example manual save mode usage

For this example, assume a radar is triggering the edgertronic camera. The delay from the event until the trigger occurs is 700 ms. Further assume you want 100ms capture before the event and 100 ms capture after the event. Since the 700ms latency is larger than the 100ms post-event you want to capture, there will be 600ms of video after the duration of interest. Those 600ms do not need to be saved, thus use CAMAPI selective_save() to specify the starting and ending frames to save.

Camera settings

Key camera settings:

  • Frame rate: 1000fps
  • Pre-trigger buffer: 800ms (100ms of pre-trigger plus the 700ms latency)
  • Post-trigger buffer: 0ms

The first frame captured is 800ms * 1000 fps = -800. Remember all pre-trigger frames have a negative value. The last frame captured is 0, the trigger frame.

From the above, we can calculate the frames of interest (those frames 100 ms before the event and 100ms after the event).

  • First frame: -800, 100ms before the event, which is 800ms before the trigger.
  • Last frame: -600, wanting 200ms of video captured at 1000fps, that is a total of 200 frames.

Configure the camera by invoking the CAMAPI run() method passing in your requested settings.

Triggering

Triggering can be done either using a radar detector contact closure connected to the camera's remote trigger input or using CAMAPI trigger() method. If you use the trigger() method, consider passing in a user parameter (perhaps the detected speed of the ball) and/or filename.

Camera monitoring

At this point, the only way to monitor changes in camera state is by polling the camera using the CAMAPI get_camstatus() method. The controlling external device should be monitoring the camera for changes in any of the following:

  • Camera capture state
  • Camera save state
  • Camera status

In python, it could be something like:

import hcamapi, time
cam = hcamapi.HCamapi("10.11.12.13")

status = cam.get_camstatus()
saving = False

while True:
   time.sleep(1)
   new_status = cam.get_camstatus()
   if saving and new_status.get('save_state') == CAMAPI_SAVE_STATE_IDLE:
        saving |= process_save_complete(cam, new_status)
   if new_status.get('active_buffer') > status.get('active_buffer'):
       saving = process_capture_complete(cam, new_status, saving)
   if new_status.get('flags') != status.get('flags'):
        process_camera_status_change(cam, new_status)
   status = new_status

The above using polling, so you have to ask yourself if it is possible to miss something important because it occurs and clears in less than the polling time. For example, if you were monitoring if the camera capture state is triggered, you could miss this even as the camera could be become trigger and then fill the post trigger buffer and again change state in less than the polling time. The logic shown above will work independent of the sleep time. The caveat is that by monitoring the flags, you could miss an error indication if the error self corrects in less than the poll time. Such an example could be the camera storage becoming full and then more space becoming available due to the controlling device deleting a saved video after it was copied to long term storage.

Processing capture state changes

Once the CAMAPI run() method is invoked, the camera increments the active_buffer (really the capture count), the state switches to filling-pre-trigger-buffer and the camera starts filling the pre-trigger buffer with video frames. If no trigger occurs when the pre-trigger buffer is full, the camera switches to pre-trigger-buffer-full state and the camera continues capturing video frames overwriting the oldest frame until a trigger occurs.

Once a trigger occurs, the camera switches to the filling-post-trigger-buffer and the camera write frames to the post-trigger buffer until the buffer is full. After the buffer is full several things happen:

  • All information about the just captured video is recorded and associated with the new entry in the captured video queue.
  • The camera locates an empty buffer in DDR3. If one is not available, the camera stops the capture and switches to the buffers-full-trigger-disabled state.
  • If a DDR3 buffer is available, the camera state switches to filling-pre-trigger-buffer and the camera starts storing videos frames in the pre-trigger portion of the buffer in the DDR3.
  • The CAMAPI get_camstatus() method returned dictionary unsaved_count entry is incremented.
  • A new entry is added to the CAMAPI get_captured_video_info() returned dictionary (and the unsaved_count in that returned dictionary is incremented as well).

Processing a new capture consists of monitoring a change in get_camstatus() active_buffer (really capture count), and once detected, the CAMAPI get_captured_video_info() method should be invoked to keep the external controlling device's cached list of available captured videos up to date.

In python, it could be something like:

def process_capture_complete(cam, status, saving):
    '''Return True if the camera starts saving a captured video.'''
    global vidque # dictionary keys are trigger times as integers
    saving = False
    vids = cam.get_captured_video_info()
    vidque_modified = False
    for key in vids.keys():
        vid = vids.get(key)
        if type(vid) is dict:
            tt = int(vid.get('trigger_time'))
            if vidque.get(tt) == None:
                vidque_modified = True
                vidque[tt] = vid
    if vidque_modifed:
        saving = controller_handle_new_videos(saving)
    return saving

Processing save state changes

In manual save mode, saving a video file is initiated by invoking the CAMAPI selective_save() method. Any segment in any buffer in the captured video queue can be saved to a video file. Once a save is in progress, the camera will indicate a save has completed

It is also possible for a save to be interrupted, due to storage full condition. In this case, the camera will:

In python, it would be something like:

def process_save_complete(cam, status):

Processing status changes

In python, it would be something like:

def process_camera_status_change(cam, status):

Random implementation notes

  1. At some point the camera may support a WebSocket to notify clients when a change in camera state occurs by sending CAMAPI get_camstatus() responses when the camera's state changes. The name of the file just saved could also be included in the camera status information. This would save the controlling external device from having to poll CAMAPI get_camstatus() to detect was a capture just finished or a save just finished. I mention this now as the camera control software is implemented to support asynchronous notifications, but the version of the lighttpd web server doesn't support WebSockets.
  2. The idea of implementing a queue of requested saves was considered and discarded. The client controlling the camera needs to monitor when a save completes and control what occurs next.