Mobius Labs provides next-generation, state-of-the-art, industry leading Computer Vision technology for edge devices and on-premise solutions. We build technology that makes devices truly visually intelligent. With our on-premise SDK, anyone can process vast amounts of images directly on their local machines on their premises. This enables independence, data privacy, security, smooth integration and control over the data flow.

Our current suite of SDK features offers large scale image and video tagging and keywording and facial recognition and search. For images, our SDK also provides aesthetic and quality score and similarity search. And for videos, our SDK also provides automatic highlights and scene changes. With many new features currently in development, Mobius Vision SDK is on its way to become a one stop shop for all of the industries’ state-of-the-art Visual AI solutions.

What makes Mobius Vision SDK truly powerful is that it allows users to be able to personalize it for their needs. Not only Mobius Vision SDK provides pre-trained AI models out-of-the-box for myriad existing use cases, it also enables users to build their own custom AI models with incredible ease, using their own data to fit any of their niche use cases.

In the following sections, you will find how each of the different modules within the Mobius Vision SDK work. Please note that this documentation only discusses the modules and SDK features that are included in your lisence. Please contact us if you are interested in additional features offered by the Mobius SDK.

First, let us run through the software and hardware requirements and setup of the Mobius Vision SDK.

To install the Mobius On-Premise SDK, you have to follow a few steps as explained here.

We provide our solution as a combination of a python package (wheel) and a Dockerfile. Using the Dockerfile allows you to build a Docker image with everything you need to run the Mobius Vision SDK. To simplify things, we also provide a docker-compose file that takes care of building the image and running it with the correct environment variables.

The access to a zipped folder with all necessary files will be delivered to you in a shipping email.

Requirements

The hardware and software requirements for the SDK differ depending on the type of server to be used (CPU or GPU).

Hardware

For the CPU version of the SDK you need:

• Intel Haswell CPU or newer (AVX2 support necessary)

For the GPU version of the SDK you need:

• Nvidia GPU of one of the following generations: Maxwell, Pascal, Volta, Turing, Ada Lovelace (Lovelace)

AMD and Intel GPUs are not supported.

A minimum of 16GB of RAM and 50 GB Disk space. Depending on the features used and the images added to the database this may increase.

Software

In order to successfully install the Mobius Vision On-Premise SDK, the following software requirements have to be met:

• GNU/Linux x86_64 with kernel version > 3.10
• docker >= 1.12 for CPU version, 19.03 or higher for GPU version
• docker-compose >= 1.28.0 (optional, but recommended)

MacOS or Windows as a host system is not supported.

Additional Software for the GPU Version

To use a GPU, the following additional software requirements have to be met:

• docker >= 19.03
• Nvidia Drivers >= 530.41.03
• nvidia-docker2 >= 2.6.0 (for nvidia-container-toolkit)

Docker Installation

These are the installation steps to install Docker on a Ubuntu based system. Steps 2 and 3 are not strictly required, but we recommend this set-up in order to prevent running the Docker container with sudo.

If you already have Docker and docker-compose installed, you can skip these steps.

1. Install the Docker Container Environment (CE) https://docs.docker.com/install/linux/docker-ce/ubuntu/

2. Add your user to the docker group.

 sudo usermod -aG docker $USER

3. Log out and log back in so that your group membership is re-evaluated.

4. Install docker-compose:

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Verify that the installed versions are equal or newer than the requirements listed above:

docker --version
docker-compose --version

Additional Steps for GPU Version

To use the GPU version of the Mobius Vision SDK you need to have nvidia-docker2. You can install it by instructions from https://github.com/NVIDIA/nvidia-docker or our instructions below.

If you already have nvidia-docker2 installed, you can skip this step.

5. Add the nvidia-docker repo to apt, as explained here: https://nvidia.github.io/nvidia-docker/

6. Install the nvidia-docker2 package and reload the Docker daemon configuration.

 sudo apt-get install nvidia-docker2
 sudo service docker restart

Verify that the installed versions are equal or newer than the requirements listed above:

nvidia-smi
nvidia-docker version

Additional Steps for migration from filesystem to PostgreSQL

When delivering a new version of the SDK to you, we will tell you if this step is required.

Older versions of the SDK used a filesystem for storage purposes. New versions use PostgreSQL database. There's a way to migrate (copy) all your data from filesystem to PostgreSQL.

Start the new SDK versions with

docker-compose up

And run the migration script with

docker-compose exec mobius_sdk migrate_data

Customer Training Interface (CTI)

In case your license includes the Customer Training Interface (CTI), please unpack the corresponding zip file you got, change to that directory and import the required Docker images using the following commands:

docker image load --input cti_frontend.tar
docker image load --input cti_backend.tar

Verify that the images were imported:

docker image ls

Include the external IP of your server in the ALLOWED_ORIGINS environment variable to allow connections to the user interface from outside of the server ('CORS policy'):

export ALLOWED_ORIGINS="http://<external_ip_of_server>:8080"

You can also permanently change this variable in the docker-compose file.

Please note that the containers are only imported here and not actually started. Do not start them yourself. They are started automatically after the SDK is started using the docker-compose up command (see next section).

Running the SDK

The SDK can be started using the following commands. The first start may take a few minutes as the docker container is then automatically built using the provided Dockerfile.

unzip mobius_sdk.zip
cd mobius_sdk
docker-compose up -d

After a while, the HTTP REST API of the SDK will be usable at localhost:5000.

In case the Custom Training Interface (CTI) is included in your license, it will be started automatically afterwards and is then available at http://<ip_of_your_server>:8080. The default username is user and the default password user as well. Additional users can be created after logging in with the username admin and the password admin.

You can verify that the docker container is running with:

docker container ps

Stopping the SDK

You can stop the SDK by executing the following command in the same directory:

docker-compose down

Configuring the Setup

Optionally, the following variables can be changed in the docker-compose file before it is executed to adapt the setup to your needs:

• SDK_PORT: port to access the API (default: 5000)
• CTI_PORT: port to access the CTI (default: 8080)
• NUM_WORKERS: number of workers for parallel processing, see note below (default: 20)
• MOBIUS_TOKEN: token to verify the validity of a particular SDK according to the license agreement (default: already set to the one included in your license)
• CUDA_VISIBLE_DEVICES: define which GPUs are used in case multiple GPUs are available (default: all)

NOTE: NUM_WORKERS should be carefully adjusted according to the features in the SDK (the more features are shipped in the SDK, the lower this value should be) and the available hardware (the more cores are available, the higher this value can be). We usually recommend a value between 5 and 50 for this environment variable.

The following environment variables are available for the Custom Training Interface (CTI):

• ALLOWED_ORIGINS: set this to the external IP of the server to prevent misuse of the CTI backend or set it to "*" to disable this security mechanism (not recommended) (default: http://localhost:8080)

• POSTGRES_PASSWORD and JWT_SECRET_KEY: random strings used for additional security (default: random passwords)

• MAX_VIDEO_UPLOAD_SIZE: set this value to the define the maximum size of uploadable video in MB in the CTI (default: 10000)

• VIDEO_PREDICTION_SDK_TIMEOUT: set this to define the period in seconds which CTI should wait for a video to be processed. After this time the CTI will throw error (default is 36000).

Usage of Docker Volumes

By modifying the docker-compose file, the volumes to be used for the user data can be changed, too:

• mobius_vision_data: docker volume used to store user data (metadata, indexes etc.)
• mobius_vision_redis: docker volume used to store redis data (database used for task status and scheduling)

You can also mount a local drive or folder to the container for faster predictions or uploads of images and videos (see the path parameter on those endpoints). In the volumes section of the mobius_sdk service in the docker-compose file, add <path_on_host>:<path_on_container>:<flags> where <path_on_host> is the full path to the directory to be mounted, and <path_on_container> is the point at which it will be mounted. <path_on_container> can either be a fully qualified path, in other words beginning with /, or it can be a relative path. If it is a relative path, it is interpreted as relative to a configurable base path which defaults to /external. It is recommended to keep this default to ensure there are no conflicts with Linux or Docker system files. <flags> can be any Docker volume mount flags, but ro (for read-only within the container) is strongly recommended.

For example, include /mnt/nfs/image_archive:/external/image_archive:ro in the docker-compose file, and then add a path parameter on requests like follows: "path": "image_archive/image0001.jpg".

Checking the SDK Status

The Mobius SDK does not have a standard endpoint to check the availability of the module. However, it can be quite easily checked by passing a query image or video for prediction.

Image prediction

Simple example for calling image prediction with a query image image.jpg with default parameters.

 curl "http://127.0.0.1:5000/image/predict" \
 -X POST \
 -F "data=@./image.jpg" \
 -F params='{}'

If the SDK is running properly and the image file can be read in the preprocessing, the SDK returns a 200 response with the status success.

Video prediction

Simple example for calling video prediction with a query videovideo.mp4 with default parameters.

 curl "http://127.0.0.1:5000/video/predict" \
 -X POST \
 -F "data=@./video.mp4" \
 -F params='{}'

If the SDK is running properly and the video file can be read in the preprocessing, the SDK returns a 200 response with the status success.

Accessing logs

In case of errors, logs may help us to fix the problem faster.

You can access them by running:

docker-compose logs

Image predictions are a core functionality of the Mobius on-premise SDK. All modules for images that have been shipped with your SDK can be queried for prediction output with the predict endpoint.

Pre-trained Modules and Train Endpoints

Most modules are pre-trained and can be used out of the box; module-dependent parameters can be used to customize the modules to your use case. Some modules need to be trained first in order to be used (e.g., customized training).

Please refer to the corresponding module description section in the sidebar (or with the links in the parameter description) to learn more on how to implement workflows for the train endpoints.

Input Parameters

This endpoint comes with a range of parameters to customize the behaviour of this functionality. modules parameter is used to pass an array specifying which modules to predict with. The parameters are grouped with the relevant module and submodule. You can find detailed descriptions of the meaning of the parameters in the explanation section of each module.

The path and url parameters may be used to specify a data file(image file or downloaded pkl feature file) on the local system or a remote host, respectively, instead of including an image file in the request form data. Only one of a data file, the path parameter, and the url parameter may be specified on single request.

Parallel Processing

To get maximum performance out of the SDK run multiple requests at the same time. The difference between parallel and sequential processing could be quite dramatic. For example, it takes 17 seconds to process 1 000 images in parallel mode and 144 seconds in sequential mode (times could be different on your instance and your set of features).

Here is an example code in Python that can be used to process images in parallel.

import requests, json
import threading
from concurrent.futures import ThreadPoolExecutor

images = ['./image.jpg', './image2.jpg', './image3.jpg']
host = '127.0.0.1:5000'
params = {}

def predict_on_image(path, params, host):
    with open(path, 'rb') as image:
        r = requests.post(
            'http://%s/image/predict'%(host),
            files={'data': image},
            data={'params': json.dumps(params)}
        )
        output = r.json()
        return output

with ThreadPoolExecutor(max_workers=20) as executor:
    def worker_func(path):
        return predict_on_image(path, params, host)

    results = list(zip(images, executor.map(worker_func, images)))

To predict the standard concepts, i.e. 10 000 pre-trained concepts that come with the Mobius Vision SDK, on an image, use the /image/predict endpoint and pass tags/standard_concepts as the module name in the modules list.

Confidence Scores and Thresholding

Our SDK returns a confidence score with each concept. The confidence score indicates the degree of confidence of our artificial intelligence of the term being a match to the visual content of the image. A value of 1.0 would indicate the AI is very confident that the concept is present in the image, while a score of 0.0 indicates that the model is certain the concept is not present. The results are sorted by confidence scores in descending order (highest to lowest), and outputs with a confidence score below 0.55 are removed from the results list by default.

NOTE: In the default settings, the number of results of the standard concepts module typically varies between 5 and 50 concepts per image.

The parameter confidence_threshold can be set by the user to customize the lowest confidence level below which the results are filtered out.

Top-N Selection

For some applications it might be useful to restrict the maximum number of concepts to be returned. For such use cases, the optional top_n parameter can be set to cut off the concept predictions so that only the top_n highest scored concepts are returned.

It’s also possible to get exactly N tags by combining the confidence threshold and the top_n parameter: set the confidence threshold to 0 and the top_n parameter to desired value N.

The Mobius SDK provides the ability to train any number of new concepts. This way users can define new concepts instead of using predefined set of concepts.

To predict the custom concepts use the /image/predict endpoint and add tags/custom_concepts in the modules list and specify the list of the custom concepts IDs in custom_concept_id_list under tags/custom_concepts.

A general workflow is summarized as follows:

• Add images to the image database using the endpoint /system/database/images/add
• Assign images to positive class using the endpoint /image/tags/custom_concepts/assign
• Assign images to negative class using the endpoint /image/tags/custom_concepts/assign
• Train custom concept using the endpoint /image/tags/custom_concepts/train
• Predict custom concepts using the endpoint /image/tags/custom_concepts/predict to predict concepts on the images that are already added to the image database or the general endpoint /image/predict.

Adding Images

To add images to the image database use the endpoint /system/database/images/add.

The image database is shared between different modules of the SDK so you need to add an image only once and use the same image_id for different custom concepts and even different modules.

See Image Database section to learn more about the image database.

Assigning Images

To assign images that are considered positives for particular custom concept to class positive use the endpoint /image/tags/custom_concepts/assign. We also recommend using parallel processing to add images.

Do the same for negative images. Negative images are optional but we highly recommend adding negative images to achieve better performance.

We assume that the user knows the basics of how to curate sets of image samples, and what is meant by assigning a positive or negative class to an image. If you are not sure please reach out to your delivery manager at the Mobius team as data curation guideline documentation can be provided.

Training

Once images are assigned to a custom concept use the endpoint /image/tags/custom_concepts/train to train it.

The training phase might take some time depending on the number of images that are assigned to positive and negative classes. A part of the response of the train endpoint is a unique task_id of the training task. Use the endpoint /system/tasks/status/get to check the progress of training. The status can be either in_queue, ongoing, success or error. Use this endpoint to find out when the training is completed (status is success or error) and the custom concept can be used for prediction.

Predict

There are two ways to predict custom concept:

1. Use the endpoint /image/tags/custom_concepts/predict to predict concepts on the images that are already added to the image database. It's the fastest way to get predictions for custom concepts because all required information is already extracted from the images. This way you can process thousands of images per second.

2. Use the general endpoint /image/predict to get predictions from an image.

The second option is much slower especially if you need to process the same image multiple times. We advice to add all your images to the image database of the SDK first and then use the endpoint /image/tags/custom_concepts/predict.

Custom Concept Management

The SDK offers some utility endpoints to manage custom concepts. You can perform the following actions:

• Delete a custom concept
• Get the list of all custom concept IDs
• Upload a custom concept
• Download a custom concept

NOTE: Please be aware that using delete is irreversible unless you have a backup of the SDK data volume or custom concept is downloaded.

Improving a Custom Concept

A custom concept is rarely trained perfectly after the first iteration. That's why we advise to do multiple iterations of the training.

1. After each training run a custom concept on a set of unlabeled images. We call it a validation set. For quicker processing add validation set images to the image database.
2. Label images that are misclassified and assign them to their corresponding classes.
3. Train the custom concept.
4. Repeat until performance is acceptable.
5. Test final version of the custom concept on a new set of images: a test set. A test set is supposed to be sampled the same way as a validation set. Make sure that images in a test set do not belong to a validation set or to the images that were used for training. If performance on a test set is as desired, then you have a custom concept that is ready to be used in the production.

Face Recognition module of the Mobius Vision SDK returns the following information from an image:

• List of faces in an image. ° Bounding Box for each face to specify where it is located in the input image. The output format specifies the respective edge coordinates in the image: {left, lower, right, upper}
• Identity of the person, if it is already registered in the identity database, along with the group that the person belongs to.
• Detected gender of the person, male or female.
• Estimated age of the person, in years.

This is also illustrated in the following image.

All of this information is accessed by using the image/predict endpoint and specifying face_rec ognition under the module key in the parameters object sent with the request.

In order to train the Mobius SDK to recognize any number of faces desired, a user simply needs to add the faces to the identity database. Please see the Identity Database section to learn how that is done and managed.

Calling /image/face_recognition/identities/predict with the following payload. The face recognition in this endpoint performs faster because the prediction is done on images already added to the database.

A common usecase for this would be running face recognition on images in the database after updating the identities. This will try to identify people in the images using the updated identities.

You need to pass it the image ids where to identify people and the groups to use for identification.

image_id_list - specify a list of images where you want to identify the people

group_id_list - specify a list of groups where you added identities

Face clustering feature allows you to find unidentified / improperly identified faces in the image database

Adding images

Before clustering may be used, a set of images should be added to the image database. This can be done by using endpoint /system/database/images/add. Parameter partition_id may be specified to indicate that particular set of images belong to partition. Only faces from same partition will be clustered. If not specified, images are added to default partition. The use_for_search parameter has to be set to True.

Clustering endpoint

Clustering endpoint /image/face_recognition/cluster may be called to cluster all faces with certain partition_id. Clusters are analyzed to find an unknown faces or faces that look like 2 or more identities within certain partition_id.

As clustering is a long operation, a part of the response of the cluster endpoint is a unique task_id of the cluster task. Use the endpoint /system/tasks/status/get to check the progress of task. The status can be either in_queue, ongoing, success or error. Use this endpoint to find out when the clustering is completed (status is success or error) and the custom detector can be used for prediction.

This endpoint is only supported in the Postgres version. Faces that were added to the filesystem version and then migrated to Postgres will be ignored.

Input Parameters

Clustering endpoint comes with a range of parameters to customize the behaviour of this functionality. partition_id parameter is used to specify a partition of the image database (should correspond with partition_id used when adding images). If not specified, faces from default partition will be clustered. top_n parameter specifies a number of clusters of faces returned group_id specifies a group of identities used for face identification (should correspond with group of identity database) min_num_faces specifies the minimum number of faces in a cluster. If the cluster has fewer faces than this, it will be discarded.

Result

Result of clustering task will be presented in /system/tasks/status/get after task is completed. Response will include unknown_identities attribute, holding a list of unidentified / improperly identified clusters. Each cluster has a list of faces in it, and each face has these attributes: image_id - identifier of the image, where this face was found bounding_box - coordinates of the face on the image identity and distance - optional attributes, only represented when some identity was matched with the face.
Each cluster will also contain faces_found which is the number of faces inside this cluster.

The Search module allows you to perform different types of search. Here is the list of search options that are available to you:

• Identity search

A general workflow of the search is summarized as follows:

• Add images to the image database using the endpoint /system/database/images/add
• Train the search module /image/search/train
• Do identity search using the endpoint /image/predict.

It might be desired to add more images to the search module after training. The workflow for that procedure is following:

• Add new images to the image database using the endpoint /system/database/images/add. This procedure by itself doesn’t add images to the search module because indexes need to be updated as well.
• Update the search module using the train endpoint /image/search/train. This procedure will add newly added images to the existing indexes. If you want to retrain indexes from scratch use parameter retrain.
• Do identity search using the endpoint /image/predict.

To add images to the image database use the endpoint /system/database/images/add.

The image database is shared between different modules of the SDK so you can add an image only once and use the same image_id for different modules that use the image database.

For image to be considered as part of the search module, you need to set use_for_search param to True. By setting update_index to True, the image will be added immediately into the search module, otherwise you need to update the search module. See Image Database chapter to learn more about the image database.

Once images are added to the image database use the endpoint /image/search/train to train search module.

The training phase might take some time. A part of the response of the train endpoint is a unique task_id of the training task. Use the endpoint /system/tasks/status/get to check the progress of training. The status can be either in_queue, ongoing, success or error. Use this endpoint to find out when the training is completed (status is success or error) and the similarity search can be used.

The parameter retrain can be used to force the search module to retrain indexes from scratch. By default, its value is set to false.

To update the indexes without starting from scratch, you can add or remove images dynamically. Simply set the update_index parameter to true when adding or removing images from the image database.

Use the endpoint /image/predict to perform the search. You can specify which type of search you want to perform in the modules key of the parameters. See descriptions below for more information of specific types of search that are available to you.

There are a few settings for the search module.

The endpoint /image/search/settings/get can be used to get the current state of the search module settings.

The endpoint /image/search/settings/set can be used to set the state of the search module settings.

identity_prob can be used to manage trade-offs between speed and accuracy of the identity search. The value is between 0 and 1. Lower values means faster search and higher values means more accurate predictions. We recommend to use value 0.1 but you can always adjust it using the endpoint /image/search/settings/set.

The identity search module allows to search for the images with the same people as in the query image. It can be useful if you need to find all the images where a particular person appears. You don’t even need to know the name of the person, one good photo of the face is sufficient.

Identity search results are obtained using the endpoint /image/predict by passing in the search/identity parameter under the modules key.

The parameter top_n can be used to control the number of results that will be returned. The default value is set to 100, meaning only the top 100 closest matches will be returned.

The custom concept search module allows to search for images using the name of any available custom concept as input. By using this feature we can quickly find the best matching images in the database for a trained custom concept.

Concept search results are obtained using the endpoint /image/search/concept_search.

The parameter top_n can be used to control the number of results that will be returned. The default value is set to 100, meaning only the top 100 closest matches will be returned for each custom concept name.

A major upgrade to the SDK may change how the search data in the database is represented, in which case a reset of the relevant database is necessary. Typically this is handled automatically, but in some cases it may be needed to do so manually. Your Mobius representative will tell you when you need to do this, as well as what additional steps - if any - need to be undertaken.

Resetting an index is also useful if you wish to erase the database and start again. Deleting each item from the database with the associated delete endpoint and then call that database's reset_default endpoint and all traces of the old data will be removed.

Remove from identities database Reset identities database index

Remove from facial expressions database Reset facial expressions database index

Remove from image database Reset image database index

The Mobius Vision SDK provides various AI modules that can extract a variety of information from an input video.

The Mobius Vision SDK is shipped with a number of modules for video processing. All modules for video that have been shipped with your SDK can be queried for prediction output with the /video/predict endpoint.

One important thing to note when processing videos with the Mobius SDK is that since videos are typically large files, it takes some time for the SDK to process the whole video. Therefore, the /video/predict call only returns a task_id and not predictions immediately. Use the endpoint /system/tasks/status/get to check the progress of video processing or use a webhook to be notified when the analysis is finished using the notification_url parameter and then get the results using this endpoint. The task status can be either in_queue, ongoing, success or error. If the status is success then the status response will also contain the results of video processing. If the status is error check the field message to get the name of the error.

A video can be seen as a sequence of images, which are called frames. Video content is typically recorded at 24 to 60 frames per second (fps). The Mobius Vision SDK processes videos on a frame by frame basis. By default, we do not use the original frame rate of the video as the processing time grows linearly with the number of frames that are processed, and content typically does not change too rapidly. Usually we use 3 frames per second for predictions, which can be changed if needed. In the rare case that frame-accurate results are required, you can use the value -1 for the extraction_fps parameter to analyze every single frame of the video.

There are a few important terms that a user needs to understand in order to be able to fully utilize the Mobius SDK's video processing. When a user calls the /video/predict endpoint, the SDK returns predictions at four levels: segment, subsegment, frame-level and video-level. These are described in the following in more detail.

Segment-level: A Segment is a collection of frames where the scene-content does not change substantially. The SDK uses an intelligent shot detector which partitions the video into a set of semantically meaningful segments. Each segment contains predictions along with the corresponding confidence scores. A segment is sometimes also referred to as 'shot' and the start and end times of the segments can be seen as shot-boundaries.

Subsegment level: To allow for more accurate localization concepts (in time), the SDK further splits video segments into fixed-length subsegments. At the default frame extraction rate of 3 fps, the subsegments are around 2.5 seconds long. Each subsegment contains predictions along with the corresponding confidence scores. This can be used to get more fine-granular prediction results in a certain shot. If the extraction_fps is set to -1 and the subsegment_length to 1, there will be exactly one subsegment per original frame of the video.

Frame level: To allow for tracking identities in the video, the SDK returns each identity and its face bounding box in each extracted frame of video (3 fps by default). Frame level prediction is not available in fast mode and the face recognition module should be available for video prediction to detect the faces for frame_level output. Frame level just contains frames which at least one identity detected on it, so each item in frame_level has a timestamp to indicate when it is started and when is ended.

Video-level: Video-level predictions consist of all predictions that appear in at least one segment of the video. For each video-level prediction, the SDK returns two values: a confidence score and a value that reflects how long the prediction appears in the video, referred to as duration . As such, the video-level predictions along with their confidence scores and durations can be seen as a way of summarizing the video.

If the face recognition module is enabled for video prediction, each of the four levels contain the identities of the recognized faces in the face_recognition field. A preview thumbnail of a recognized identity can be accessed by using the endpoint /system/database/identities/reference_image/get if it is already in the face identity database, or endpoint video/thumbnails/unknown_identity/download if it was tagged as an unknown person (recognized by person_id similar to [Unknown X]).

There are a few general video processing parameters that are applied to every module:

• extraction_fps specifies the number of frames per second (FPS) extracted from the video for processing. Default value is 3. Higher rate will take longer for processing, but will analyse finer transitions in the video giving more information. If set to -1, every single original frame will be analyzed. This is very resource intensive and can result in large prediction outputs, so only use this if really necessary.

• pooling_window_length defines the number of frames over which smoothing of the predictions will be applied.

• video_level_enabled is the flag that enables the output of video-level predictions. Default value is true.

• segment_level_enabled is the flag that enables the output of segment-level predictions. Default value is true.

• subsegment_level_enabled is the flag that enables the output of subsegment-level predictions. Default value is true.

• frame_level_enabled is the flag that enables the output of frame-level predictions. Default value is false.

• thumbnails_enabled is the flag that enables the saving of previews for segment and subsegment. The previews can be accessed later by using the endpoint /video/thumbnails/subsegment/download and endpoint /video/thumbnails/segment/download. Default value is false.

• unknown_face_enabled is the flag that enables the saving of previews for unknown identities faces if the face is not found in any listed identities groups in the request. The previews can be accessed later by using the endpoint video/thumbnails/unknown_identity/download. Default value is false.

• subsegment_length defines the length of each subsegment as the number of frames in regards to the extraction_fps framerate. If the extraction_fps is set to 10 and the subsegment_length is 20, then each subsegment is 2 seconds long (20 frames / 10 fps = 2s). This length should be shorter than the expected length of a shot, but still larger than 1 to allow the 'pooling' / smoothing of prediction results over multiple extracted frames.

• path and url parameters may be used to specify a file on the local system or a remote host, respectively. Only one of a data file, the path parameter, and the url parameter may be specified on single request.

• notification_url is the HTTP POST endpoint to notify when processing the video is finished. For authorization and encryption of the request, you can set these environment variables when running the SDK:

  NOTIF_AUTH=Bearer
  NOTIF_TOKEN=myToken
  NOTIF_SECRET=StrongSecret

  If NOTIF_SECRET is not set, the data will not be encrypted.

The payload of the notification is a JSON object with a status field and the task_id of the video analysis task. This task_id can then be used to retrieve the full analysis output from the SDK using the /system/tasks/status/get endpoint. After the results and face thumbnails were retrieved, they can also be deleted from the SDK using the /video/results/delete endpoint to free up the storage.

The payload also contains a repetition of the metadata of the analyzed video (also found in the analysis output) that can for example be used for usage tracking by keeping track of the video lengths.

The webhook should return a success response (e.g. status code 200) to the SDK.

Here is an example code in Python that can be used to implement the webhook handler with flask.

from flask import Flask, jsonify, request
import jwt

app = Flask(__name__)
secret = 'StrongSecret'

@app.route('/webhook', methods=['POST'])
def webhook():
    headers = request.headers
    auth_info = headers.get('Authorization')
    # Authorization with token
    # ...

    payload = request.get_json()['data']
    if secret:
        payload = jwt.decode(payload, secret, algorithms=["HS256"])

    task_id = payload.get('task_id')
    task_status = payload.get('status')

    return jsonify({
        'status': 'success',
    })

if __name__ == "__main__":
    app.run(host='0.0.0.0', port='5000')

To predict the standard concepts, i.e. 10 000 pre-trained concepts that come with the Mobius Vision SDK, on a video, use the general /video/predict endpoint and pass tags/standard_concepts as the module name in the modules list.

Confidence Scores and Thresholding

Our SDK returns a confidence score with each keyword. The confidence score indicates the degree of confidence of our artificial intelligence of the term being a match to the visual content. A value of 1.0 would indicate the AI is very confident that the concept is present in the image, while a score of 0.0 indicates that the model is certain the concept is not present. The results are sorted by confidence scores in descending order (highest to lowest), and outputs with a confidence score below 0.55 are removed from the results list by default.

The parameter confidence_threshold can be set by the user to customize the lowest confidence level below which the results are filtered out.

Top-N Selection

For some applications it might be useful to restrict the maximum number of tags to be returned. For such use cases, the optional top_n parameter can be set to cut off the concept predictions so that only the top_n highest scored tags are returned.

It’s also possible to get exactly N tags by combining the confidence threshold and the top_n parameter: set the confidence threshold to 0 and the top_n parameter to desired value N.

Configuration File

The on-premise SDK comes with a default configuration file, named default. The user can fully customize these settings and create their own custom configuration files.

The name of the configuration file is passed during prediction time in the prediction call in order to specify which setting to use (see /tags/standard_concepts/config_name in the parameters of the /video/predict endpoint).

The naming of the configuration files enables the usage of multiple configuration files in the same SDK.

There are three endpoints to manage configuration files for the standard concepts on video:

• Download a configuration file
• Upload a configuration file
• Delete a configuration file

To get more information about the configuration file refer to chapter Configuration Files for Standard Concepts.

The Mobius SDK provides the ability to train any number of new concepts. This way users can define new concepts instead of using predefined sets of concepts.

For the information on how to train custom concepts refer to the chapter Custom Concepts.

To retrieve the custom concepts, use the /video/predict endpoint add tags/custom_concepts as the module name in the modules list and add custom concept IDs to the custom_concept_id_list under tags/custom_concepts.

Following diagram gives a meaningful illustration of the results returned by the Custom Concepts module.

Mobius Vision SDK can recognize 11,000 celebrities out-of-the-box. To identify faces on videos, use the /video/predict endpoint and pass face_recognition as the module name in the modules list.

In addition, the Mobius SDK also allows users to add any number of new faces to customize the SDK for their use case. Please see the Identity Database section for information on how to add new faces.

To recognize cartoon characters in videos, do not add them to the identity database as it is only trained for human faces. Rather use custom concepts here.

Here is an overview of how the results are returned on videos.

It is possible to add newly detected unknown identities inside the video to the identity database. For this purpose set store_identities inside the face_recognition/identities section of /video/predict endpoint and set the identities group_id which you like to add new found identities into. Newly added identities info will be returned inside the info section of the video result. The user can tune below params to select which identities should be added.

• min_quality: Minimum quality of face for adding it into the identities group (default: 0.8)

• min_frames: Minimum visited times of the unknown face inside the video (default: 15, equivalent of 3 seconds). In case of increasing the fps of video processing or using the fast_mode this params should be increased or decreased. The default processing fps of video is 3.

Using low values for min_quality and min_frames causes more low quality faces to be added into the identities database. To avoid this, min_quality should be around 0.7 or higher.

This feature allows to obtain highlight scores for video segments, which can be used to identify the most important parts of a video. This can be very useful for example in order to create a summary of a video that can be shown if someone is browsing through a video database, or to identify the highlights in a recording of unedited video.

For illustration purposes we show an example of a video that can benefit from highlighting. Someone is standing on the side of a motocross race, waiting for the race to start.

Our highlight detector is able to identify the point where the race actually starts, as identified by the spike of the highlight scores towards the end of the clip.

Below is the section of the video that corresponds to the identified highlight. Note that depending on the length and type of video, there can be numerous (or none) highlights detected by our highlights detector.

To extract hightlights from videos, use the /video/predict endpoint and pass highlights or highlights/general_highlights as the module name in the modules list.

This section contains endpoints that are not features by itself, but which are very useful or even necessary for the operation of the on-premise SDK.