SDK (2.0.0)

Download OpenAPI specification:Download

Introduction

Mobius Labs’ On-Premise SDK offers large scale image and video tagging, facial recognition and search. For images, our SDK also provides aesthetic and quality score and similarity search. And for videos, our SDK also provides automatic highlights and scene changes. With many new features currently in development, Mobius Vision SDK is on its way to become a one stop shop for all of the industries’ state-of-the-art Visual AI solutions.

What makes Mobius Vision SDK truly powerful is that it allows users to be able to personalize it for their needs. Not only Mobius Vision SDK provides pre-trained AI models out-of-the-box for myriad existing use cases, it also enables users to build their own custom AI models with incredible ease, using their own data to fit any of their niche use cases.

In the following sections, you will find how each of the different modules within the Mobius Vision SDK work. Please note that this documentation only discusses the modules and SDK features that are included in your lisence. Please contact us if you are interested in additional features offered by the Mobius SDK.

First, let us run through the software and hardware requirements and setup of the Mobius Vision SDK.

Installation

To install the Mobius On-Premise SDK, you have to follow a few steps as explained here. We provide our solution as a combination of a python package (wheel) and a Dockerfile. Using the Dockerfile allows you to build a Docker image with everything you need to run the Mobius Vision SDK. The access to a zipped folder with all necessary files will be delivered to you in a shipping email.

Requirements

The hardware and software requirements for the SDK differ depending on the type of server to be used (CPU or GPU).

In order to successfully install the Mobius Vision On-Premise SDK, the following software requirements have to be met:

  • GNU/Linux x86_64 with kernel version > 3.10
  • docker >= 1.12

We support the following hardware generations for Intel CPUs: Haswell or higher (AVX2 support necessary).

Additional Requirements for the GPU Version

In order to successfully install the Mobius Vision On-Premise SDK for GPU, the following software requirements have to be met:

  • docker >= 1.12
  • Nvidia Drivers >= 410.48
  • nvidia-docker2

We support the following hardware generations for GPUs: Kepler, Maxwell, Pascal, Volta, Turing.

Docker Installation

CPU Version

These are the installation steps if you have a CPU build of the Mobius Vision SDK. Steps 2 and 3 are not strictly required, but we recommend this set-up in order to prevent running the Docker container with sudo.

If you already have Docker installed, you can skip the docker installation step.

  1. Install the Docker CE https://docs.docker.com/install/linux/docker-ce/ubuntu/

  2. Add your user to the docker group.

 sudo usermod -aG docker $USER
  1. Log out and log back in so that your group membership is re-evaluated.

GPU Version

To use the GPU version of the Mobius Vision SDK you need to have nvidia-docker2. You can install it by instructions from https://github.com/NVIDIA/nvidia-docker or our instructions below.

If you already have nvidia-docker2 installed, you can skip this step.

  1. Install the Docker CE https://docs.docker.com/install/linux/docker-ce/ubuntu/
  1. Add the nvidia-docker repo to apt, as explained here: https://nvidia.github.io/nvidia-docker/

  2. Install the nvidia-docker2 package and reload the Docker daemon configuration.

 sudo apt-get install nvidia-docker2
 sudo pkill -SIGHUP dockerd
  1. Add your user to the docker group.
 sudo usermod -aG docker $USER
  1. Log out and log back in so that your group membership is re-evaluated.

Building the Docker Image

The Docker image can be built directly from the zipped folder that has been shipped to you.

CPU Version

Build the Mobius Vision SDK Docker image.

 unzip mobius_sdk.zip
 cd mobius_sdk
 docker build -t mobius_labs/mobius_sdk:latest -f Dockerfile .

GPU Version

Build the Mobius Vision SDK Docker image.

 unzip mobius_sdk.zip
 cd mobius_sdk
 nvidia-docker build -t mobius_labs/mobius_sdk:latest -f Dockerfile .

Verification of the Docker Image

To check that the Docker image was built successfully, you can run the following command:

 docker images

In the list of all Docker images, mobius_sdk should appear like this:

 REPOSITORY               TAG           IMAGE ID            CREATED             SIZE
 mobius_labs/mobius_sdk   latest        d4a9c41f3496        2 minutes ago      6.69GB

Now you are ready to run the SDK.

Running

Once the SDK and all required software packages are installed you can start by running the Docker image.

Running the SDK

The command for running the SDK differs depending on whether docker or nvidia-docker was used to install the SDK.

CPU Version

 docker run --init -v mobius_vision_data:/data -v mobius_vision_redis:/var/lib/redis -p 5000:5000 \
             -e NUM_WORKERS="20" -e MOBIUS_TOKEN="<your_token>" \
             -it mobius_labs/mobius_sdk:latest

GPU Version

 nvidia-docker run --init -v mobius_vision_data:/data -v mobius_vision_redis:/var/lib/redis -p 5000:5000 \
                   -e CUDA_VISIBLE_DEVICES="0" -e NUM_WORKERS="20" -e MOBIUS_TOKEN="<your_token>" \
                   -it mobius_labs/mobius_sdk:latest

where the parameters have the following meaning:

  • mobius_vision_data : the name of the docker volume that will be used to store user data
  • mobius_vision_redis : the name of the docker volume that will be used to store redis data
  • CUDA_VISIBLE_DEVICES : (GPU only) environment variable that controls which CUDA-capable devices the SDK can use for processing. Multiple devices can be specified as well if separated by a comma
  • NUM_WORKERS : number of workers for parallel processing
  • MOBIUS_TOKEN : token provided during the delivery for Mobius to verify the validity of a particular SDK according to the license agreement
  • mobius_labs/mobius_sdk:latest : name of the Docker image as listed by Docker

NOTE: NUM_WORKERS should be carefully adjusted according to the features in the SDK (the more features are shipped in the SDK, the lower this value should be) and the available hardware (the more cores are available, the higher this value can be). We usually recommend a value between 5 and 50 for this environment variable.

Checking the SDK Status

The Mobius SDK does not have a standard endpoint to check the availability of the module. However, it can be quite easily checked by passing a query image or video for prediction.

Image prediction

Simple example for calling image prediction with a query image image.jpg with default parameters.

 curl "http://127.0.0.1:5000/image/predict" \
 -X POST \
 -F "data=@./image.jpg" \
 -F params='{}

If the SDK is running properly and the image file can be read in the preprocessing, the SDK returns a 200 response with the status success.

Video prediction

Simple example for calling video prediction with a query videovideo.mp4 with default parameters.

 curl "http://127.0.0.1:5000/video/predict" \
 -X POST \
 -F "data=@./video.mp4" \
 -F params='{}

If the SDK is running properly and the video file can be read in the preprocessing, the SDK returns a 200 response with the status success.

Predict

Image predictions are a core functionality of the Mobius on-premise SDK. All modules for images that have been shipped with your SDK can be queried for prediction output with the predict endpoint.

Pre-trained Modules and Train Endpoints

Most modules are pre-trained and can be used out of the box; module-dependent parameters can be used to customize the modules to your use case. Some modules need to be trained first in order to be used (e.g., customized training).

Please refer to the corresponding module description section in the sidebar (or with the links in the parameter description) to learn more on how to implement workflows for the train endpoints.

Input Parameters

This endpoint comes with a range of parameters to customize the behaviour of this functionality. modules parameter is used to pass an array specifying which modules to predict with. The parameters are grouped with the relevant module and submodule. You can find detailed descriptions of the meaning of the parameters in the explanation section of each module.

Parallel Processing

To get maximum performance out of the SDK run multiple requests at the same time. The difference between parallel and sequential processing could be quite dramatic. For example, it takes 17 seconds to process 1 000 images in parallel mode and 144 seconds in sequential mode (times could be different on your instance and your set of features).

Here is an example code in Python that can be used to process images in parallel.

import requests, json
import threading
from concurrent.futures import ThreadPoolExecutor

images = ['./image.jpg', './image2.jpg', './image3.jpg']
host = '127.0.0.1:5000'
params = {}

def predict_on_image(path, params, host):
    with open(path, 'rb') as image:
        r = requests.post(
            'http://%s/image/predict'%(host),
            files={'data': image},
            data={'params': json.dumps(params)}
        )
        output = r.json()
        return output

with ThreadPoolExecutor(max_workers=20) as executor:
    def worker_func(path):
        return predict_on_image(path, params, host)

    results = list(zip(images, executor.map(worker_func, images)))

Predict on Image

post /image/predict

Localhost

http://127.0.0.1:5000/image/predict

Endpoint for predictions on a query image with module selection and a range of optional parameters.

Request Body schema: multipart/form-data
params
object (image_predict_params)
data
required
string <binary>

Image file

Responses

200

OK

400

Error

Request samples

Content type
multipart/form-data
Copy
{
  "params": {
    "modules": [
      "tags/standard_concepts",
      "face_recognition/bounding_boxes",
      "aesthetics/stock_score",
      "aesthetics/custom_styles",
      "tags/custom_concepts",
      "face_recognition/identities",
      "face_recognition/gender",
      "aesthetics/quality_score",
      "search/similarity",
      "face_recognition/age"
    ],
    "tags": {
      "standard_concepts": {
        "confidence_threshold": 0.5,
        "top_n": 100,
        "categories_enabled": true
      },
      "custom_concepts": {
        "custom_concept_id_list": [
          "leather jacket",
          "spiderman"
        ]
      }
    },
    "search": {
      "similarity": {
        "top_n": 5,
        "filter": [
          "identities"
        ]
      }
    },
    "aesthetics": {
      "custom_styles": {
        "custom_style_id_list": [
          "minimalism",
          "still life"
        ]
      }
    },
    "face_recognition": {
      "identities": {
        "group_id_list": [
          "default",
          "test"
        ]
      }
    }
  },
  "data": "..."
}

Response samples

Content type
application/json
Copy
Expand all Collapse all
{
  • "tags":
    {
    },
  • "aesthetics":
    {
    },
  • "face_recognition":
    [
    ],
  • "search":
    {
    },
  • "status": "success",
  • "params":
    {
    }
}

Tags

Standard Concepts

To predict the standard concepts, i.e. 10 000 pre-trained concepts that come with the Mobius Vision SDK, on an image, use the /image/predict endpoint and pass tags/standard_concepts as the module name in the modules list.

Confidence Scores and Thresholding

Our SDK returns a confidence score with each concept. The confidence score indicates the degree of confidence of our artificial intelligence of the term being a match to the visual content of the image. A value of 1.0 would indicate the AI is very confident that the concept is present in the image, while a score of 0.0 indicates that the model is certain the concept is not present. The results are sorted by confidence scores in descending order (highest to lowest), and outputs with a confidence score below 0.55 are removed from the results list by default.

NOTE: In the default settings, the number of results of the standard concepts module typically varies between 5 and 50 concepts per image.

The parameter confidence_threshold can be set by the user to customize the lowest confidence level below which the results are filtered out.

Top-N Selection

For some applications it might be useful to restrict the maximum number of concepts to be returned. For such use cases, the optional top_n parameter can be set to cut off the concept predictions so that only the top_n highest scored concepts are returned.

It’s also possible to get exactly N tags by combining the confidence threshold and the top_n parameter: set the confidence threshold to 0 and the top_n parameter to desired value N. This can be achieved through the [Configuration Files] (#section/Configuration-Files-for-Standard-Concepts).

Custom Concepts

The Mobius SDK provides the ability to train any number of new concepts. This way users can define new concepts instead of using predefined set of concepts.

To predict the custom concepts use the /image/predict endpoint and add tags/custom_concepts in the modules list and specify the list of the custom concepts IDs in custom_concept_id_list under tags/custom_concepts.

A general workflow is summarized as follows:

Adding Images

To add images to the image database use the endpoint /system/database/images/add.

The image database is shared between different modules of the SDK so you need to add an image only once and use the same image_id for different custom concepts and even different modules.

See Image Database section to learn more about the image database.

Assigning Images

To assign images that are considered positives for particular custom concept to class positive use the endpoint /image/tags/custom_concepts/assign. We also recommend using [parallel processing] (#tag/ImagePredict/#tag/ParallelProcessing) to add images.

Do the same for negative images. Negative images are optional but we highly recommend adding negative images to achieve better performance.

We assume that the user knows the basics of how to curate sets of image samples, and what is meant by assigning a positive or negative class to an image. If you are not sure please reach out to your delivery manager at the Mobius team as data curation guideline documentation can be provided.

Training

Once images are assigned to a custom concept use the endpoint /image/tags/custom_concepts/train to train it.

The training phase might take some time depending on the number of images that are assigned to positive and negative classes. A part of the response of the train endpoint is a unique task_id of the training task. Use the endpoint /system/tasks/status/get to check the progress of training. The status can be either in_queue, ongoing, success or error. Use this endpoint to find out when the training is completed (status is success or error) and the custom concept can be used for prediction.

Predict

There are two ways to predict custom concept:

  1. Use the endpoint /image/tags/custom_concepts/predict to predict concepts on the images that are already added to the image database. It's the fastest way to get predictions for custom concepts because all required information is already extracted from the images. This way you can process thousands of images per second.

  2. Use the general endpoint /image/predict to get predictions from an image.

The second option is much slower especially if you need to process the same image multiple times. We advice to add all your images to the image database of the SDK first and then use the endpoint /image/tags/custom_concepts/predict.

Custom Concept Management

The SDK offers some utility endpoints to manage custom concepts. You can perform the following actions:

NOTE: Please be aware that using delete is irreversible unless you have a backup of the SDK data volume or custom concept is downloaded.

Improving a Custom Concept

A custom concept is rarely trained perfectly after the first iteration. That's why we advise to do multiple iterations of the training.

  1. After each training run a custom concept on a set of unlabeled images. We call it a validation set. For quicker processing add validation set images to the image database.
  2. Label images that are misclassified and assign them to their corresponding classes.
  3. Train the custom concept.
  4. Repeat until performance is acceptable.
  5. Test final version of the custom concept on a new set of images: a test set. A test set is supposed to be sampled the same way as a validation set. Make sure that images in a test set do not belong to a validation set or to the images that were used for training. If performance on a test set is as desired, then you have a custom concept that is ready to be used in the production.

Upload Configuration File for Standard Concepts

post /image/tags/standard_concepts/config/upload

Localhost

http://127.0.0.1:5000/image/tags/standard_concepts/config/upload
Request Body schema: multipart/form-data
params
object
data
required
string <binary>

Image file

Responses

200

OK

400

Error

Request samples

Content type
multipart/form-data
Copy
{
  "params": {
    "name": "test"
  },
  "data": "..."
}

Response samples

Content type
application/json
Copy
Expand all Collapse all
{
  • "status": "success"
}

Download Configuration File for Standard Concepts

post /image/tags/standard_concepts/config/download

Localhost

http://127.0.0.1:5000/image/tags/standard_concepts/config/download
Request Body schema: multipart/form-data
params
object

Responses

200

OK