# Overview

The Demographic Estimation Library is a portable C++ library to estimate the persons' demographic characteristics (age, gedner, etc).

The SDK provides wrappers in the following languages:

* C++ (native)
* C
* Python
* C# / .NET

## Release Notes

* **Version 1.0.0 (24 Jan 2024)**
   * Initial release

## Getting Started

### Hardware requirements

The SDK doesn't have any special hardware requirement:

- **CPU:** No special requirement, any modern 64 bit capable CPU (x86-64 with AVX, ARM8) is supported
- **GPU:** No special requirement
- **RAM:** 2 GB of available RAM required
- **Camera:** No special requirement, minimum resolution: 640x480.

### Software requirements

The SDK is regularly tested on the following Operating Systems:

- Windows 10
- Ubuntu 22.4
- Mac OS 12 Monterey
- iOS 14
- Android 12

### 3rd Party Licenses

While the SDK is released under a proprietary license, the following Open-Source projects where used in it with their respective licenses:

- OpenCV - [3 clause BSD](https://opencv.org/license/)
- Tensorflow - [Apache License 2.0](https://github.com/tensorflow/tensorflow/blob/master/LICENSE)
- Protobuf - [3 clause BSD](https://github.com/protocolbuffers/protobuf/blob/master/LICENSE)
- zlib - [zlib license](https://www.zlib.net/zlib_license.html)
- minizip-ng - [zlib license](https://github.com/zlib-ng/minizip-ng/blob/master/LICENSE)
- stlab - [Boost Software License 1.0](https://github.com/stlab/libraries/blob/main/LICENSE)
- docopt.cpp - [MIT License](https://github.com/docopt/docopt.cpp/blob/master/LICENSE-MIT)
- pybind11 - [3 clause BSD](https://github.com/pybind/pybind11/blob/master/LICENSE)
- fmtlib - [MIT License](https://github.com/fmtlib/fmt/blob/master/LICENSE.rst)

### Dependencies

The public C++ API hides all the implementation details from the user, and it only depends on the C++17 Standard Library.
It also provides a binary compatible interface, making it possible to change the underlying implementation
without the need of recompilation of the user code.

### Installation

#### C++

Extract the SDK contents, include the headers from the `include` folder and link `libDemographicEstimationLibrary` to your C++ project.

#### Python

The python version of the SDK can be installed with pip:

```bash
$ pip install realeyes.demographic_estimation
```

#### C# / .NET

The .NET version of the SDK can be installed via NuGet:

```bash
$ dotnet add package Realeyes.DemographicEstimation
```

## Usage

### C++

The main entry point of this library is the `del::DemographicEstimator` class.

After a **estimator** object was constructed, the user can call the `del::DemographicEstimator::detectFaces()` function to get the faces
for a frame of a video or other frame source.

The resulting `del::Face` objects can be used to call `del::DemographicEstimator::estimate()` to get the classifiers estimations of the face.
The results returned in `del::Output` structs.

`del::DemographicEstimator::detectFaces()` and `del::DemographicEstimator::estimate()` has two versions both are non-blocking async calls, one is returning **std::future** the other
is calling the callback on completion. After one call a subsequent call is possible without waiting for the result.

For the frame data, the user must construct a `del::ImageHeader` object and pass that to `del::DemographicEstimator::detectFaces()`.
The frame data must outlive this object since it is a non-owning view of the data, but it only needs to be valid during the
`del::DemographicEstimator::detectFaces()` call, the library will copy the frame data internally.

The following example shows the basic usage of the library using OpenCV for loading images and feeding them to the estimator:

```cpp
#include "demographicestimator.h"

#include <opencv2/core.hpp>
#include <opencv2/imgcodecs.hpp>

#include <iostream>

int main()
{
   del::DemographicEstimator estimator("model/model.realZ");

   cv::Mat image = cv::imread("image.jpg");

   std::vector<del::Face> faces = estimator.detectFaces({image.ptr(), image.cols, image.rows, static_cast<int>(image.step1()), del::ImageFormat::BGR}).get();

   for (const del::Face& face: faces) {
       auto results = estimator.estimate(face).get();
       del::BoundingBox bbox = face.boundingBox();
       std::cout << "User at (" << bbox.x << ", " << bbox.y << ", " << bbox.width << ", " << bbox.height << ") has the following estimated characteristics:" << std::endl;

       for (const del::Output& result: results) {
           std::cout << "  " << result.name << " = ";
           if (result.type == del::OutputType::AGE || result.type == del::OutputType::AGE_UNCERTAINTY)
               std::cout << std::get<float>(result.value) << std::endl;
           else
               std::cout << ((std::get<del::Gender>(result.value) == del::Gender::FEMALE) ? "FEMALE" : "MALE") << std::endl;
       }
       std::cout << std::endl;
   }
   return 0;
}
```

### Python

The main entry point of this library is the `realeyes.demographic_estimation.DemographicEstimator` class.

After a **estimator** object was constructed, the user can call the `realeyes.demographic_estimation.DemographicEstimator.detect_faces()` function to get the faces
for a frame of a video or other frame source.

The resulting `realeyes.demographic_estimation.Face` objects can be used to call `realeyes.demographic_estimation.DemographicEstimator.estimate()` to get the classifiers estimations of the face.

The following example shows the basic usage of the library using OpenCV for loading images and feeding them to the estimator:

```python
import realeyes.demographic_estimation as del
import cv2

estimator = del.DemographicEstimator('model/model.realZ')

image = cv2.imread('image.jpg')[:, :, ::-1]  # opencv reads BGR we need RGB

faces = estimator.detect_faces(image)

for face in faces:
    print(f'User at {face.bounding_box()} has the following estimated characteristics:')
    results = estimator.estimate(face)
    for result in results:
        print(f'  {result.name} = {result.value} ')
```

### C# / .NET

The main entry point of this library is the `DemographicEstimator` class.

After an **estimator** object is constructed, you can call the `DetectFacesAsync()` method to detect faces
in a frame. The method returns a `Task<FaceList>` allowing for asynchronous, non-blocking operation.

The resulting `Face` objects can be used to call `EstimateAsync()` to get demographic estimations.
Both methods support concurrent execution - you can start multiple operations in parallel without waiting for results.

The following example demonstrates parallel processing of multiple frames using the async interface:

```csharp
using Realeyes.DemographicEstimation;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

class Program
{
    static async Task Main(string[] args)
    {
        using var estimator = new DemographicEstimator("model/model.realZ");

        // Load an image (example using raw byte array)
        byte[] imageData = LoadImageData("image.jpg");
        var imageHeader = new ImageHeader(imageData, 1920, 1080, 1920 * 3, ImageFormat.RGB);

        // Detect faces asynchronously
        await using var faces = await estimator.DetectFacesAsync(imageHeader);

        Console.WriteLine($"Detected {faces.Count} face(s)");

        // Start parallel demographic estimation for all faces
        var estimationTasks = faces.Select(face => estimator.EstimateAsync(face)).ToList();

        // Check concurrent calculations in flight
        Console.WriteLine($"Running {estimator.ConcurrentCalculations} concurrent calculations");

        // Wait for all estimations to complete
        var demographics = await Task.WhenAll(estimationTasks);

        // Process results
        for (int i = 0; i < faces.Count; i++)
        {
            var face = faces[i];
            var result = demographics[i];

            Console.WriteLine($"Face at ({face.BoundingBox.X}, {face.BoundingBox.Y}, " +
                            $"{face.BoundingBox.Width}, {face.BoundingBox.Height}):");

            if (result.Age.HasValue)
                Console.WriteLine($"  Age: {result.Age.Value:F1}");

            if (result.Gender.HasValue)
                Console.WriteLine($"  Gender: {result.Gender.Value}");

            if (result.AgeUncertainty.HasValue)
                Console.WriteLine($"  Age Uncertainty: {result.AgeUncertainty.Value:F2}");
        }
    }

    static byte[] LoadImageData(string path)
    {
        // Implementation depends on your image loading library
        throw new NotImplementedException();
    }
}
```

## Results {#target-to-face-specs}

The **Face** objects consist of the following members:

* **bounding_box:** Bounding box of the detected face (left, top, width height).
* **confidence:** Confidence of the detection ([0,1] more is better).
* **landmarks:** 5 landmarks from the face:
   1. left eye
   2. right eye
   3. nose (tip)
   4. left mouth corner
   5. right mouth corner

   ![landmarks](landmarks.png)

## 3rd party face detector

It is possible to calculate the embedding of a face which was detected with a different library. One can create
`Face` object by specifying the source image and the landmarks.