Rekognition Face Detection and Matching

The Nomad Media face detection pipeline uses Amazon Rekognition together with DynamoDB to detect faces in assets, cluster similar faces, and build a person identity model over time.

DynamoDB Tables

PersonFace

• externalFaceId — String, Primary Key
• personId — String

Person

• id — String, Primary Key
• name — String
• namingPriority — Number, Sort Key
• automatchCount — Number
• probableMatchCount — Number

If name is defined, namingPriority must be set to 0.

Face

• externalFaceId — String, Primary Key
• id — String
• boundingBox — String
• assetId — String
• previewImageUrl — String
• matchType — String
• personId — String

UnassignedFace

• externalFaceId — String, Primary Key
• count — Number, Sort Key

When a face is inserted into the Face table with no personId, it is also inserted into UnassignedFace with count = 0. Each time a face-to-face match is inserted into FaceFace, the count is incremented for both matched faces in UnassignedFace. When a person is created, their associated faces are removed from UnassignedFace and added to PersonFace.

Workflow

1. Rekognition Lambda

1. Detects faces in the asset using Rekognition.
2. Inserts each face into the Face table with an empty personId.
3. Inserts each face into the UnassignedFace table with count = 0.

2. Search Image Faces Lambda

1. For each detected face, calls Rekognition to find similar faces.
2. Loops through similar faces:
   • Verifies both the original and similar face exist in the Face table; inserts if missing.
   • If similarity exceeds application/rekognitionSettings/probableMatchSimilarityThreshold:
     • Inserts into the FaceFace table.
     • Increments count in UnassignedFace for both FaceId and SimilarFaceId.
     • If the similar face already has a personId, queues the original face for person ID assignment.
3. For each face queued for person ID assignment:
   • Updates personId and matchType in the Face table.
   • Removes from UnassignedFace.
   • Updates automatchCount and probableMatchCount in the Person table.
   • Adds externalFaceId and personId to the PersonFace table.
   • Sets namingPriority = automatchCount + probableMatchCount if Person.Name is blank; sets to 0 if Person.Name is populated.

3. Create Persons Lambda

Runs on a scheduled basis (frequency TBD).

1. Queries UnassignedFace for the face with the highest count.
2. Queries FaceFace for all entries matching that externalFaceId.
3. Creates a new Person with namingPriority = number of FaceFace matches.
4. Updates the Face table for all matched faces to assign the new personId.
5. Deletes matched faces from UnassignedFace.
6. Inserts matched faces into PersonFace.