# Overview

The Face Verification Library is a portable C++ library for face verification.

The SDK also has wrappers in the following languages:

• C
• Python
• C# (.NET)

• Version 1.5.0 (12 Jun 2024)

  • Improved performance on ARM CPUs
  • Cleaner .Net API
  • New public C API

• Version 1.4.0 (7 Feb 2024)

  • Experimental .Net support

• Version 1.3.0 (9 Jun 2023)

  • Add model config version 2 support

• Version 1.2.0 (9 Jun 2023)

  • Add support for AES encryption

• Version 1.1.0 (3 Apr 2023)

  • Add support for 3rd party face detectors

• Version 1.0.0 (1 Marc 2023)

  • Initial release

The SDK doesn't have any special hardware requirement:

• CPU: No special requirement, any modern 64 bit capable CPU (x86-64 with AVX, ARM8) is supported
• GPU: No special requirement
• RAM: 2 GB of available RAM required
• Camera: No special requirement, minimum resolution: 640x480.

The SDK is regularly tested on the following Operating Systems:

• Windows 10
• Ubuntu 22.4
• Mac OS 12 Monterey
• iOS 14
• Android 12

While the SDK is released under a proprietary license, the following Open-Source projects where used in it with their respective licenses:

• OpenCV - 3 clause BSD
• Tensorflow - Apache License 2.0
• Protobuf - 3 clause BSD
• zlib - zlib license
• minizip-ng - zlib license
• stlab - Boost Software License 1.0
• docopt.cpp - MIT License
• pybind11 - 3 clause BSD
• fmtlib - MIT License

The public C++ API hides all the implementation details from the user, and it only depends on the C++17 Standard Library. It also provides a binary compatible interface, making it possible to change the underlying implementation without the need of recompilation of the user code.

Extract the SDK contents, include the headers from the include folder and link libFaceVerificationLibrary to your C++ project.

Extract the SDK contents, include faceverifier_c.h from the include folder and link libFaceVerificationLibrary to your C project.

The python version of the SDK can be installed with pip:

$ pip install realeyes.face_verification

The .NET version of the SDK can be installed via NuGet:

$ dotnet add package Realeyes.FaceVerification

The main entry point of this library is the fvl::FaceVerifier class.

After a verifier object was constructed, the user can call the fvl::FaceVerifier::detectFaces() function to get the faces for a frame of a video or other frame source.

The resulting fvl::Face objects can be used to call fvl::FaceVerifier::embedFace() to get the embedding of the face.

Two embedding can be comapred with the call to fvl::FaceVerifier::compareFaces or manually using any of the standard similarity/distance functions over the embeddings.

fvl::FaceVerifier::detectFaces() and fvl::FaceVerifier::embedFace() has two versions both are non-blocking async calls, one is returning std::future the other is calling the callback on completion. After one call a subsequent call is possible without waiting for the result.

For the frame data, the user must construct a fvl::ImageHeader object and pass that to fvl::FaceVerifier::detectFaces(). The frame data must outlive this object since it is a non-owning view of the data, but it only needs to be valid during the fvl::FaceVerifier::detectFaces() call, the library will copy the frame data internally.

The following example shows the basic usage of the library using OpenCV for loading images and feeding them to the verifier:

#include "faceverifier.h"

#include <opencv2/core.hpp>
#include <opencv2/imgcodec.hpp>

#include <iostream>

int main()
{
   fvl::FaceVerifier verifier("model/model.realZ");

   cv::Mat image1 = cv::imread("image1.jpg");
   cv::Mat image2 = cv::imread("image2.jpg");

   std::vector<fvl::Face> faces1 = verifier.detectFaces({image1.ptr(), image1.cols, image1.rows, static_cast<int>(image1.step1()), fvl::ImageFormat::BGR}).get();
   std::vector<fvl::Face> faces2 = verifier.detectFaces({image2.ptr(), image2.cols, image2.rows, static_cast<int>(image2.step1()), fvl::ImageFormat::BGR}).get();



   std::vector<std::vector<float>> embeddings1, embeddings2;
   for (const Face& face: faces1)
       embeddings1.push_back(verifier.embedFace(face).get());
   for (const Face& face: faces2)
       embeddings2.push_back(verifier.embedFace(face).get());

   for (size_t i = 0; i < embeddings1.size(); ++i)
       for (size_t j = 0; j < embeddings2.size(); ++j)
           if (verifier.compare_faces(embeddings1[i], embeddings2[j]).similarity > 0.3) {
               fvl::BoundingBox bbox1 = faces1[i].bounding_box();
               fvl::BoundingBox bbox2 = faces2[j].bounding_box();
               std::cout << "(" << bbox1.x << ", " << bbox1.y << ", " << bbox1.width << ", " << bbox1.height << ") from image 1 and";
               std::cout << "(" << bbox2.x << ", " << bbox2.y << ", " << bbox2.width << ", " << bbox2.height << ") from image 2 are the same people";
               std::cout << std::endl;
           }

   return 0;
}

The main entry point of this library is the realeyes.face_verification.FaceVerifier class.

After a verifier object was constructed, the user can call the realeyes.face_verification.FaceVerifier.detect_faces() function to get the faces for a frame of a video or other frame source.

The resulting realeyes.face_verification.Face objects can be used to call realeyes.face_verification.FaceVerifier.embed_face() to get the embedding of the face.

Two embedding can be comapred with the call to realeyes.face_verification.FaceVerifier.compare_faces() or manually using any of the standard similarity/distance functions over the embeddings.

The following example shows the basic usage of the library using OpenCV for loading images and feeding them to the verifier:

import realeyes.face_verification as fvl
import cv2

verifier = fvl.FaceVerifier('model/model.realZ')

image1 = cv2.imread('image1.jpg')[:, :, ::-1]  # opencv reads BGR we need RGB
image2 = cv2.imread('image2.jpg')[:, :, ::-1]  # opencv reads BGR we need RGB

faces1 = verifier.detect_faces(image1)
faces2 = verifier.detect_faces(image2)

embeddings1 = [verifier.embed_face(face) for face in faces1]
embeddings2 = [verifier.embed_face(face) for face in faces2]

for i, e1 in enumerate(embeddings1):
    for j, e2 in enumerate(embeddings2):
        if verifier.compare_faces(e1, e2).similarity > 0.3:
            print(f'{faces1[i]} from image 1 and {faces2[j]} from image 2 are the same person!')

The main entry point of this library is the Realeyes.FaceVerification.FaceVerifier class.

After a verifier object was constructed, the user can call DetectFacesAsync() to get the faces for a frame of a video or other frame source.

The resulting Face objects can be used to call EmbedFaceAsync() to get the embedding of the face.

Two embeddings can be compared with the call to CompareFaces() or manually using any of the standard similarity/distance functions over the embeddings.

The following example shows the basic usage of the library using OpenCV for loading images and feeding them to the verifier:

using Realeyes.FaceVerification;
using OpenCvSharp;

using var verifier = new FaceVerifier("model/model.realZ");

using var image1 = Cv2.ImRead("image1.jpg");
using var image2 = Cv2.ImRead("image2.jpg");

// Convert OpenCV Mat to byte array
var imageData1 = image1.ToBytes();
var imageData2 = image2.ToBytes();

var header1 = new ImageHeader(imageData1, image1.Width, image1.Height,
    image1.Width * image1.Channels(), ImageFormat.BGR);
var header2 = new ImageHeader(imageData2, image2.Width, image2.Height,
    image2.Width * image2.Channels(), ImageFormat.BGR);

// Detect faces in parallel
var detectTask1 = verifier.DetectFacesAsync(header1);
var detectTask2 = verifier.DetectFacesAsync(header2);

using var faces1 = await detectTask1;
using var faces2 = await detectTask2;

// Embed all faces in parallel
var embeddingTasks = new List<Task<float[]>>();
foreach (var face in faces1)
    embeddingTasks.Add(verifier.EmbedFaceAsync(face));
foreach (var face in faces2)
    embeddingTasks.Add(verifier.EmbedFaceAsync(face));

var allEmbeddings = await Task.WhenAll(embeddingTasks);

var embeddings1 = allEmbeddings.Take(faces1.Count).ToArray();
var embeddings2 = allEmbeddings.Skip(faces1.Count).ToArray();

// Compare embeddings
for (int i = 0; i < embeddings1.Length; i++)
{
    for (int j = 0; j < embeddings2.Length; j++)
    {
        var match = verifier.CompareFaces(embeddings1[i], embeddings2[j]);
        if (match.Similarity > 0.3f)
        {
            var bbox1 = faces1[i].BoundingBox;
            var bbox2 = faces2[j].BoundingBox;
            Console.WriteLine($"({bbox1.X}, {bbox1.Y}, {bbox1.Width}, {bbox1.Height}) from image 1 and " +
                              $"({bbox2.X}, {bbox2.Y}, {bbox2.Width}, {bbox2.Height}) from image 2 are the same person!");
        }
    }
}

The Face objects consist of the following members:

• bounding_box: Bounding box of the detected face (left, top, width height).
• confidence: Confidence of the detection ([0,1] more is better).

• landmarks: 5 landmarks from the face:

  1. left eye
  2. right eye
  3. nose (tip)
  4. left mouth corner
  5. right mouth corner

  

It is possible to calculate the embedding of a face which was detected with a different library. One can create Face object by specifying the source image and the landmarks.