SYSTEMS AND METHODS FOR RETRIEVING OBJECTS VIA PROMPT-BASED TRACKING

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001]This patent application claims priority from, and incorporates by reference the entire disclosure of, U.S. Provisional Patent Application No. 63/570,156 filed on Mar. 26, 2024.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

[0002]This invention was made with government support under 1946391 awarded by the National Science Foundation. The government has certain rights in the invention.

TECHNICAL FIELD

[0003]The present disclosure relates generally to prompt-based tracking and more particularly, but not by way of limitation, to systems and methods for retrieving objects via prompt-based tracking.

BACKGROUND

[0004]This section provides background information to facilitate a better understanding of the various aspects of the disclosure. The statements in this section of this document are to be read in this light, and not as admissions of prior art.

[0005]Multiple Object Tracking (MOT) is a challenging task that requires locating and identifying multiple objects in a video sequence. Existing MOT methods often rely on models trained with predefined object classes to perform tracking. However, these methods are limited by the availability and diversity of object categories and annotations.

SUMMARY OF THE INVENTION

[0006]This summary is provided to introduce a selection of concepts that are further described below in the Detailed Description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it to be used as an aid in limiting the scope of the claimed subject matter.

[0007]In an embodiment, the present disclosure pertains to a method for tracking an object with a textual description. In certain embodiments, the method includes building a third-order tensor. In some embodiments, the third-order tensor includes an image from a video, an object trajectory based, at least in part, on a previous image of the video, and text. In certain embodiments, the method further includes extracting a visual feature from an image region of the image and the object trajectory, determining an attention matrix based, at least in part, on at least one of the image region, the object trajectory, or the text, correlating the image region, the object trajectory, and the text, generating a context-aware object representation, incorporating the context-aware object representation with the visual feature, decoding an object bounding box and score from the context-aware object representation, tracking the object across at least one frame of the video, and predicting a trajectory of the object in the video.

[0008]In an addition embodiment, the present disclosure pertains to a system for tracking an object with a textual description. In some embodiments, the system includes memory and at least on processor coupled to the memory. In certain embodiments, the processor configured to implement a method that includes building a third-order tensor. In some embodiments, the third-order tensor includes an image from a video, an object trajectory based, at least in part, on a previous image of the video, and text. In certain embodiments, the method further includes extracting a visual feature from an image region of the image and the object trajectory, determining an attention matrix based, at least in part, on at least one of the image region, the object trajectory, or the text, correlating the image region, the object trajectory, and the text, generating a context-aware object representation, incorporating the context-aware object representation with the visual feature, decoding an object bounding box and score from the context-aware object representation, tracking the object across at least one frame of the video, and predicting a trajectory of the object in the video.

[0009]In a further embodiment, the present disclosure pertains to a computer-program product having a non-transitory computer-usable medium having computer-readable program code embodied therein. In certain embodiments, the computer-readable program code is adapted to be executed to implement a method for tracking an object. In certain embodiments, the method includes building a third-order tensor. In some embodiments, the third-order tensor includes an image from a video, an object trajectory based, at least in part, on a previous image of the video, and text. In certain embodiments, the method further includes extracting a visual feature from an image region of the image and the object trajectory, determining an attention matrix based, at least in part, on at least one of the image region, the object trajectory, or the text, correlating the image region, the object trajectory, and the text, generating a context-aware object representation, incorporating the context-aware object representation with the visual feature, decoding an object bounding box and score from the context-aware object representation, tracking the object across at least one frame of the video, and predicting a trajectory of the object in the video.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010]A more complete understanding of the subject matter of the present disclosure may be obtained by reference to the following Detailed Description when taken in conjunction with the accompanying Drawings wherein:

[0011]FIG. 1 illustrates an example method for tracking an object with a textual description according to aspects of the present disclosure.

[0012]FIGS. 2A-2B illustrate an example of the responsive Type-to-Track. A user provides a video sequence and a prompting request. During tracking, the system is able to discriminate appearance attributes to track the target subjects accordingly and iteratively responds to the user's tracking request.

[0013]FIGS. 3A-3B illustrate an example sequence and annotations in a dataset. FIG. 3A illustrates the MOT17 subset sample in both action and appearance. FIG. 3B illustrates the TAO subset samples with captions.

[0014]FIGS. 4A-4B illustrate an example word cloud showing words in a language description. FIG. 4A shows the MOT17 subset and FIG. 4B shows the TAO subset.

DETAILED DESCRIPTION

[0016]It is to be understood that the following disclosure provides many different embodiments, or examples, for implementing different features of various embodiments. Specific examples of components and arrangements are described below to simplify the disclosure. These are, of course, merely examples and are not intended to be limiting. The section headings used herein are for organizational purposes and are not to be construed as limiting the subject matter described.

[0017]Language supervision is a technique that leverages natural language descriptions to provide additional guidance and contextual information to computer vision models. By using image-text pairs as inputs, language supervision can help computer vision models learn a richer set of visual concepts and transfer them to various downstream tasks. Tracking objects based on semantic and descriptive text inputs is challenging and requires integrating visual and textual information. Unlike traditional object tracking algorithms that rely on deep visual features representing colors, shapes, and textures, tracking objects based on semantic and descriptive input involves semantic understanding and the matching of the textual description to the objects present in the scene.

[0018]Models need to deal with significant challenges, including, but not limited to, class-agnostic object initialization and adapting to scene changes, such as object appearance and disappearance. These challenges affect both the detection and the tracking components of the model. Additionally, the detection component needs to accurately associate the textual description with the corresponding object in each frame. Tracking objects based on semantic and descriptive text inputs also require class-agnostic object initialization, the problem of finding and identifying objects of described types in the text query, without relying on predefined object classes or labels. Various methods have been proposed to approach this problem, including deep learning techniques. However, despite these advances, there is still room for improvement in intuitiveness and responsiveness. One potential way to improve object tracking in videos is to incorporate user input into the tracking process.

[0019]Traditional Visual Object Tracking (VOT) methods typically require users to manually select objects in a video by points, bounding boxes, or trained object detectors. Thus, the task that combines responsive typing input to guide the tracking of objects in videos, called Grounded Object Tracking, allows for more intuitive and conversational tracking, as users can simply type in the name or description of the object they wish to track. Most of the recent methods for the Grounded Single Object Tracking task are not class-agnostic, meaning they require prior knowledge of the object. GTI and TransVLT need to input the initial bounding box, while TrackFormer needs the pre-defined category. The operation used to fuse visual and textual features is concatenation which can only support prompts describing a single object.

[0020]This disclosure presents, inter alia, a novel framework for Grounded Multiple Object Tracking (GMOT) which is retrieving and tracking objects with text initialization. A new transformer-based eMbed-ENcoDE-extRact framework (MENDER) is introduced with third-order tensor decomposition as the first efficient approach for this task. Advantageously, the proposed MENDER model reduces the computational complexity of third-order correlations by designing an efficient attention method that scales quadratically with reference to the input sizes.

[0021]In contrast to the methods above, the MOT approach, MENDER, formulates third-order attention to adaptively focus on many targets, and it is an efficient single-stage and class-agnostic framework. Moreover, even handling three types of input tokens, i.e. text, images, and tracklets, the models presented herein reduce the computational complexity of three-dimension transformer structures, which have cubic time and space complexity, by designing an efficient attention method that scales quadratically with respect to image token size, the numbers of text tokens and tracklets.

Methods for Tracking an Object

[0022]In view of the above, in an embodiment illustrated in FIG. 1, a method for tracking an object with a textual description (100) is disclosed. In some embodiments, the method may include building a tensor (102). In certain embodiments, the tensor may be, for example, a third-order tensor. In some embodiments, the third-order tensor may include an image, an object trajectory, and text. In some embodiments, the image may be from, for example, a video. In some embodiments, the object trajectory is based, at least in part, on a previous image of the video. In certain embodiments, the text may be inputted by a user. In some embodiments, the text is in the form of a prompt which may be provided by, for example, the user.

[0023]In some embodiments, the method may also include extracting a visual feature from an image region of the image and the object trajectory (104) and determining an attention matrix (106). In certain embodiments, the attention matrix may be based, at least in part, the image region, the object trajectory, the text, or combinations thereof. In certain embodiments, the attention matrix may include, for example, an image region attention matrix, an object trajectory matrix, a text matrix, or a combination of the same and like. In some embodiments, the attention matrix may be a third-order attention matrix. In certain embodiments, the attention matrix may adaptively focus on a plurality of objects.

[0024]In certain embodiments, the method may also include correlating the image region, the object trajectory, and the text (108) and generating a context-aware object representation (110). In some embodiments, the context-aware object representation may be capable of preserving identity information while adapting to changes in position of the object.

[0025]In some embodiments, the method may include incorporating the context-aware object representation with the visual feature (112), decoding an object bounding box and score from the context-aware object representation (114), tracking the object across at least one frame of the video (116), and predicting a trajectory of the object in the video (118).

[0026]In certain embodiments, the method may include optimizing a parameter associated with the extracting step (104), the encoding step, the decoding step (114), or combinations of the same and like. In some embodiments, optimizing may utilize an algorithm. In certain embodiments, the algorithm may include a deep learning algorithm or similar machine learning technique and/or artificial intelligence algorithms.

[0027]In some embodiments, the method may also include encoding the visual feature. In some embodiments, the method may also include scaling. In certain embodiments, the scaling may be performed with a reference. In some embodiments, the reference may be based, at least in part, on an input size. In some embodiments, the scaling may include quadratic scaling. In certain embodiments, the quadratic scaling scales quadratically with respect to input size of the image. In some embodiments, the scaling may include linear scaling or combinations of linear and quadratic scaling.

[0028]In certain embodiments, the methods of the present disclosure may be implemented in the form of a system. For example, in certain embodiments, a system may include, for example, memory and at least one processor. In some embodiments, the at least one processor is coupled to the memory. In certain embodiments, the processor is configured to implement the methods as disclosed herein.

[0029]Additionally, the methods of the present disclosure may be incorporated into a medium adapted to execute code. For example, in certain embodiments, the medium may include a non-transitory computer-usable medium having computer-readable program code embodied therein. In certain embodiments, the computer-readable program code may be adapted to be executed to implement the methods of the present disclosure.

Applications and Advantages

[0030]The methods of the present disclosure have the potential to impact various fields. For example, the methods of the present disclosure may impact surveillance and robotics, where recognizing object interactions is a crucial task. The methods presented herein can improve the intuitiveness and responsiveness of tracking, making it more practical for video input support in large-language models and real-world applications like popular artificial intelligence systems.

EXAMPLE EMBODIMENTS

[0032]Reference will now be made to more specific embodiments of the present disclosure and experimental results that provide support for such embodiments. However, Applicant notes that the disclosure below is for illustrative purposes only and is not intended to limit the scope of the claimed subject matter in any way.

Type-to-Track: Retrieve any Object Via Prompt-Based Tracking

[0033]One of the recent trends in vision problems is to use natural language captions to describe the objects of interest. This approach can overcome some limitations of traditional methods that rely on bounding boxes or category annotations. This embodiment introduces a novel paradigm for Multiple Object Tracking called Type-to-Track, which allows users to track objects in videos by typing natural language descriptions. Applicant presents a new dataset for that Grounded Multiple Object Tracking task, called GroOT, that contains videos with various types of objects and their corresponding textual captions describing their appearance and action in detail. Additionally, Applicant introduces two new evaluation protocols and formulate evaluation metrics specifically for this task. Applicant develops a new efficient method that models a transformer-based eMbed-ENcoDE-extRact framework (MENDER) using the third-order tensor decomposition. The experiments in five scenarios show that Applicant's

[0034]MENDER approach outperforms another two-stage design in terms of accuracy and efficiency, up to 14.7% accuracy and 4× speed faster.

[0035]Introduction. Tracking the movement of objects in videos is a challenging task that has received significant attention in recent years. Various methods have been proposed to tackle this problem, including deep learning techniques. However, despite these advances, there is still room for improvement in intuitiveness and responsiveness. One potential way to improve object tracking in videos is to incorporate user input into the tracking process. Traditional Visual Object Tracking (VOT) methods typically require users to manually select objects in the video by points, bounding boxes, or trained object detectors. Thus, in this embodiment, Applicant introduce a new paradigm, called Type-to-Track, to this task that combines responsive typing input to guide the tracking of objects in videos. It allows for more intuitive and conversational tracking, as users can simply type in the name or description of the object they wish to track, as illustrated in FIG. 2A and FIG. 2B. Applicant's intuitive and user-friendly Type-to-Track approach has numerous potential applications, such as surveillance and object retrieval in videos.

[0036]Applicant presents a new Grounded Multiple Object Tracking dataset named GroOT that is more advanced than existing tracking datasets. GroOT contains videos with various types of multiple objects and detailed textual descriptions. It is 2× larger and more diverse than any existing datasets, and it can construct many different evaluation settings. In addition to three easy-to-construct experimental settings, Applicant proposes two new settings for prompt-based visual tracking. It brings the total number of settings to five, which will be presented below. These new experimental settings challenge existing designs and highlight the potential for further advancements.

[0037]In summary, this embodiment addresses the use of natural language to guide and assist the Multiple Object Tracking (MOT) tasks with the following contributions. First, a novel paradigm named Type-to-Track is proposed, which involves responsive and conversational typing to track any objects in videos. Second, a new GroOT dataset is introduced. It contains videos with various types of objects and their corresponding textual descriptions of 256K words describing definition, appearance, and action. Next, two new evaluation protocols that are tracking by retrieval prompts and caption prompts, and three class-agnostic tracking metrics are formulated for this problem. Finally, a new transformer-based eMbed-ENcoDE-extRact framework (MENDER) is introduced with third-order tensor decomposition as the first efficient approach for this task. Applicant's contributions in this embodiment include a novel paradigm, a rich semantic dataset, an efficient methodology, and challenging benchmarking protocols with new evaluation metrics. These contributions will be advantageous for the field of Grounded MOT by providing a valuable foundation for the development of future algorithms.

[0038]Related Work: Visual Object Tracking Datasets and Benchmarks: Datasets. To develop and train VOT models for the computer vision task of tracking objects in videos, various datasets have been created and widely used. Some of the most popular datasets for VOT are OTB, VOT, GOT, MOT challenges and BDD100K. Visual object tracking has two sub-tasks: Single Object Tracking (SOT) and Multiple Object Tracking (MOT). Table 1 shows that there is a wide variety of object tracking datasets in both types available, each with its own strengths and weaknesses. Existing datasets with NLP only support the SOT task, while Applicant's GroOT dataset supports MOT with approximately 2× larger in description size.

[0039]Benchmarks. Current benchmarks for tracking can be broadly classified into two main categories: Tracking by Bounding Box and Tracking by Natural Language, depending on the type of initialization. Previous benchmarks were limited to test videos before the emergence of deep trackers. The first publicly available benchmarks for visual tracking were OTB-2013 and OTB-2015, having 50 and 100 video sequences, respectively. GOT-10 k is a benchmark featuring 10K videos classified into 563 classes and 87 motions. TrackingNet, a subset of the object detection benchmark YT-BB, includes 31K sequences. Furthermore, there are long-term tracking benchmarks such as OxUVA and LaSOT. OxUvA spans 14 hours of video in 337 videos, having 366 object tracks. On the other hand, LaSOT is a language-assisted dataset having 1.4K sequences with 9.8K words in their captions. In addition to these benchmarks, TNL2K includes 2K video sequences for natural language-based tracking and focuses on expressing the attributes. LaSOT and TNL2K support one benchmarking setting with their provided prompts, while Applicant's GroOT dataset supports five settings. Ref-KITTI is built upon the KITTI dataset and has two categories, including car and pedestrian, while Applicant's GroOT dataset focuses on category-agnostic tracking, and outnumbers the frames and settings.

[0040]A similar task with a different nomenclature to the Grounded MOT task is Referring Video Object Segmentation (Ref-VOS), which primarily measures the overlapping area between the ground truth and prediction for a single foreground object in each caption, with less emphasis on densely tracking multiple objects over time. In contrast, Applicant's proposed Type-to-Track paradigm is distinct in its focus on responsively and conversationally typing to track any objects in videos, requiring maintaining the temporal motions of multiple objects of interest.

[0041]Grounded Object Tracking. Grounded Vision-Language Models accurately map language concepts onto visual observations by understanding both vision content and natural language. For instance, visual grounding seeks to identify the location of nouns or short phrases (such as a black hat or a blue bird) within an image. Grounded captioning can generate text descriptions and align predicted words with object regions in an image. Visual dialog enables meaningful dialogues with humans about visual content using natural, conversational language. Some visual dialog systems may incorporate referring expression recognition to resolve expressions in questions or answers.

[0042]Grounded Single Object Tracking is limited to tracking a single object with box-initialized and language-assisted methods. The GTI framework decomposes the tracking by language task into three sub-tasks: Grounding, Tracking, and Integration, and generates tubelet predictions frame-by-frame. AdaSwitcher module identifies tracking failure and switches to visual grounding for better tracking. Others introduce a unified system using attention memory and cross-attention modules with learnable semantic prototypes. Another transformer-based approach is presented including a cross-modal fusion module, task-specific heads, and a proxy token-guided fusion module.

[0043]Discussion Most existing datasets and benchmarks for object tracking are limited in their coverage and diversity of language and visual concepts. Additionally, the prompts in the existing Grounded SOT benchmarks do not contain variations in covering many objects in a single prompt, which limits the application of existing trackers in practical scenarios. To address this, Applicant presents a new dataset and benchmarking metrics to support the emerging trend of the Grounded MOT, where the goal is to align language descriptions with fine-grained regions or objects in videos.

[0046]Data Overview: Data Collection and Annotation. Existing object tracking datasets are typically designed for specific types of video scenes. To cover a diverse range of scenes, GroOT was created using official videos and bounding box annotations from the MOT17, TAO, and MOT20. The MOT17 dataset includes 14 sequences with diverse environmental conditions such as crowded scenes, varying viewpoints, and camera motion. The TAO dataset is composed of videos from seven different datasets, such as the ArgoVerse and BDD datasets containing outdoor driving scenes, while LaSOT and YFCC100M datasets include in-the-wild internet videos. Additionally, the AVA, Charades, and HACS datasets include videos depicting human-human and human-object interactions. By combining these datasets, GroOT covers multiple types of scenes and encompasses a wide range of 833 objects. This diversity allows for a wide range of object classes with captions to be included, making it an invaluable resource for training and evaluating visual grounding algorithms.

[0047]Applicant released the textual description annotations in COCO format. Specifically, a new key ‘captions’ which is a list of strings is attached to each ‘annotations’ item in the official annotation. In the MOT17 subset, Applicant attempts to maintain two types of captions for well-visible objects: one describes the appearance and the other describes the action. For example, the caption for a well-visible person might be [‘a man wearing a gray shirt’, ‘person walking on the street’] as shown in FIG. 3A. However, 10% of tracklets only have one caption type, and 3% do not have any captions due to their low visibility. The physical characteristics of a person or their personal accessories, such as their clothing, bag color, and hair color are considered to be part of their appearance. Therefore, the appearance captions include verbs ‘carrying’ or ‘holding’ to describe personal accessories. In the TAO subset, objects other than humans have one caption describing appearance, for instance, [‘a red and black scooter’]. Objects that are human have the same two types of captions as the MOT17 subset. An example is shown in FIG. 3B. These captions are consistently annotated throughout the tracklets. FIGS. 4A-4B are the word-cloud visualization of Applicant's annotations.

[0049]Class-Agnostic Evaluation Metrics. Long-tailed classification is a very challenging task in imbalanced and large-scale datasets, such as TAO. This is because it is difficult to distinguish between similar fine-grained classes, such as bus and van, due to the class hierarchy. Additionally, it is even more challenging to treat every class independently. The traditional method of evaluating tracking performance leads to inadequate benchmarking and undesired tracking results. In Applicant's Type-to-Track paradigm, the main task is not to classify objects to their correct categories but to retrieve and track the object of interest. Therefore, to alleviate the negative effect, Applicant reformulates the original per-category metrics of MOTA, IDF1, and HOTA into class-agnostic metrics:

$\begin{array}{cc}\mathrm{MOTA}=\frac{1}{❘{\mathrm{CLS}}^n❘}\stackrel{{\mathrm{CLS}}^n}{\sum _{\mathrm{cls}}}{\left(1-\frac{{\sum }_i\left({\mathrm{FN}}_t+{\mathrm{FP}}_t+{\mathrm{IDS}}_t\right)}{{\sum }_t{\mathrm{GT}}_t}\right)}_{\mathrm{cls}},\mathrm{CA}-\mathrm{MOTA}=1-\frac{{\sum }_t{\left({\mathrm{FN}}_t+{\mathrm{FP}}_t+{\mathrm{IDS}}_t\right)}_{{\mathrm{CLS}}^1}}{{\sum }_t{\left({\mathrm{GT}}_{{\mathrm{CLS}}^1}\right)}_t}& \left(1\right)\end{array}$$\begin{array}{cc}\mathrm{IDF}1=\frac{1}{❘{\mathrm{CLS}}^n❘}\stackrel{{\mathrm{CLS}}^n}{\sum _{\mathrm{cls}}}{\left(\frac{2\times \mathrm{IDTP}}{2\times \mathrm{IDTP}+\mathrm{IDFP}+\mathrm{IDFN}}\right)}_{\mathrm{cls}},\mathrm{CA}-\mathrm{IDF}1=\frac{{\left(2\times \mathrm{IDTP}\right)}_{{\mathrm{CLS}}^i}}{{\left(2\times \mathrm{IDTP}+\mathrm{IDFP}+\mathrm{IDFN}\right)}_{{\mathrm{CLS}}^1}}& \left(2\right)\end{array}$$\begin{array}{cc}\mathrm{HOTA}=\text{⁠}\frac{1}{❘{\mathrm{CLS}}^n❘}\stackrel{{\mathrm{CLS}}^n}{\sum _{\mathrm{cls}}}{\left(\sqrt{\mathrm{DetA}·\mathrm{AssA}}\right)}_{\mathrm{cls}},\mathrm{CA}-\mathrm{HOTA}=\sqrt{\left({\mathrm{DetA}}_{{\mathrm{CLS}}^1}\right)·\left({\mathrm{AssA}}_{{\mathrm{CLS}}^1}\right)}& \left(3\right)\end{array}$

where CLSⁿ is the category, set size n is reduced to 1 by combining all elements: CLSⁿ→CLS¹.

$\begin{array}{cc}{C}_t=\text{⁠}\underset{\gamma }{\mathrm{dec}}\left(\mathrm{enc}\left(I_t\right)\stackrel{\_}{\times }{\mathrm{emb}\left(P\right)}^T,\mathrm{enc}\left(I_t\right)\right)={\left\{c_i=\left(c_x,c_y,c_{\omega },c_h,c_{\mathrm{conf}}\right),❘i<M\right\}}_t& \left(4\right)\end{array}$

where (x) is an operation representing the region-prompt correlation, that will be elaborated below,

$\begin{array}{c}\mathrm{dec}\\ \gamma \end{array}\left(·,·\right)$

is an object decoder taking the similarity and the image features to decode to object locations, thresholded by a scoring parameter γ (i.e. c_conf≥γ). For simplicity, the cardinality of the set of objects |C_t|=M, implying each image token produces one region-text correlation.

[0052]Remark 1 Third-order Tensor Modeling. Since the Type-to-Track paradigm requires three input components I_t, T_t-1, and P, an auto-regressive single-stage end-to-end framework can be formulated via third-order tensor modeling.

[0053]To achieve this objective, a combination of initialization, object decoding, visual encoding, feature extraction, word embedding, and aggregation can be formulated as in Eqn. (5):

$\begin{array}{cc}{T}_t=\{\begin{array}{cc}\mathrm{initialize}\left(C_t\right)& t=0\\ \underset{\gamma }{\mathrm{dec}}(1_{D\times D\times D}{\times }_1\mathrm{enc}\left(I_t\right){\times }_2\mathrm{ext}\left(T_{t-1)}{\times }_3\mathrm{emb}\left(P\right),\mathrm{enc}\left(I_t\right)\right)& \forall t>0\end{array}& \left(5\right)\end{array}$

$\begin{array}{cc}{\theta }_{\mathrm{enc},\mathrm{ext},\mathrm{emb}}^{*}=\arg \underset{{\theta }_{\mathrm{enc},\mathrm{ext},\mathrm{emb}}}{\max }\left(l\mathrm{og}\left(\frac{\exp \left(T_{\mathrm{ijk}}\right)}{{\sum }_l^K{\sum }_n^N{\sum }_m^M\exp \left(T_{\mathrm{lnm}}\right)}\right)\right)& \left(6\right)\end{array}$

where θ denotes the network's parameters, the combination of the i^th image token, the j^th tracklet, and the k^th prompt token is the correlated triplet.

[0055]In the following, Applicant elaborates the model design for the tracking function T(I_t, T_t-1, P), named MENDER, as defined in Eqn. (5), and loss functions for the problem objective in Eqn. (6).

[0057]Remark 2 Correlation Simplification. Since both enc(⋅) and ext(⋅) are visual encoders, the region-prompt correlation can be equivalent to the tracklet-prompt correlation. Therefore, the region-tracklet-prompt correlation tensor T can be simplified to lower the computation footprint.

[0058]To design that goal, the extractor and encoder share network weights for computational efficiency:

$\begin{array}{cc}{\mathrm{ext}\left(T_{t-1}\right)}_j=\mathrm{ext}\left({\left\{{\mathrm{tr}}_j\right\}}_{t-1}\right)=\left\{{\mathrm{enc}\left(I_{t-1)}\right)}_i:c_i\mapsto {\mathrm{tr}}_j\right\},\mathrm{therefore}\left({\left(T_{:j:}\right)}_{t-1}={\left(T_{i\text{::}}\right)}_t\right):c_i\mapsto {\mathrm{tr}}_j& \left(7\right)\end{array}$

$\begin{array}{cc}{T}_t=\text{⁠}\underset{\gamma }{\mathrm{dec}}\text{⁠}\left(\left(\mathrm{enc}\left(I_t\right)\stackrel{\_}{\times }{\mathrm{ext}\left(T_{t-1}\right)}^{\top }\right)\times \left(\mathrm{ext}\left(T_{t-1}\right)\stackrel{\_}{\times }{\mathrm{emb}\left(P\right)}^{\top }\right)·\mathrm{enc}\left(I_t\right)\right)\forall t>0& \left(8\right)\end{array}$

[0059]Correlation Representations. In Applicant's approach, the correlation operation (x) is modelled by the multi-head cross-attention mechanism, as depicted in FIG. 4B. The attention matrix can be computed as:

$\begin{array}{cc}\sigma \left(X\right)\stackrel{\_}{\times }\sigma \left(Y\right)={𝒜}_{X❘Y}=\mathrm{softmax}\left(\frac{\left(\sigma \left(X\right)\times W_Q^X\right)\times {\left(\sigma \left(Y\right)\times W_K^Y\right)}^T}{\sqrt{D}}\right)& \left(9\right)\end{array}$

where X and Y tokens are one of these types: region, tracklet, or prompt. σ(⋅) is one of the operations enc(⋅), emb(⋅), or ext(⋅) as the corresponding operation to X or Y. Superscript W_Q, W_K, and W_V are the projection matrices corresponding to X or Y as in the attention mechanism.

$\begin{array}{cc}{Z}_t={𝒜}_{I❘T\times T❘P}\times \left(\mathrm{emb}\left(P\right)\times W_V^P\right)+{𝒜}_{I❘T}\times \left(\mathrm{ext}\left(T_{t-1}\right)\times W_V^T\right)& \left(10\right)\end{array}$

[0061]Object Decoder dec (⋅) utilizes context-aware features Z_t that are capable of preserving identity information while adapting to changes in position. The tracklet set T_t is defined in the auto-regressive manner to adjust to the movements of the object being tracked as in Eqn. (8). For decoding the final output at any frame, the decoder transforms the object representation by a 3-layer FFN to predict bounding boxes and confidence scores for frame t:

$\begin{array}{cc}{T}_i={\left\{{\mathrm{tr}}_j={\left({\mathrm{tr}}_x,{\mathrm{tr}}_y,{\mathrm{tr}}_{\omega },{\mathrm{tr}}_h,{\mathrm{tr}}_{\mathrm{conf}}\right)}_j\right\}}_i\stackrel{{\mathrm{tr}}_{\mathrm{conf}}\ge \gamma }{=}\mathrm{FFN}\left(Z_i+\mathrm{enc}\left(I_i\right)\right)& \left(11\right)\end{array}$

where the identification information of tracklets, represented by tr_id, is not determined directly by the FFN model. Instead, the tr_id value is set when the tracklet is first initialized and maintained till its end, similar to tracking-by-attention approaches.

[0062]Training Losses. To achieve the training objective function as in Eqn. (6), Applicant formulates the objective function into two loss functions L_I|T and L_T|P for correlation training and one loss L_GIoU for decoder training:

$\begin{array}{cc}ℒ={\gamma }_{T❘P}{L}_{T❘P}+{\gamma }_{I❘T}{L}_{I❘T}+{\gamma }_{\mathrm{GIoU}}{L}_{\mathrm{GIoU}}& \left(12\right)\end{array}$

where γ_T|P, γ_I|T, and γ_GIoU are corresponding coefficients, which are set to 0.3 by default.

[0063]Alignment Loss L_T|P is a contrastive loss, which is used to assure the alignment of the ground-truth object feature and caption pairs (T, P) which can be obtained in the dataset. There are two alignment losses used, one for all objects normalized by the number of positive prompt tokens and the other for all prompt tokens normalized by the number of positive objects. The total loss can be expressed as:

$\begin{array}{cc}{L}_{T❘P}=-\frac{1}{❘P^{+}❘}\stackrel{❘P^{+}❘}{\sum _k}\log \left(\frac{\exp \left(\mathrm{ext}{\left(T\right)}_j^{\top }\times {\mathrm{emb}\left(P\right)}_k\right)}{\underset{i}{\sum ^K}\exp \left({\mathrm{ext}\left(T\right)}_j^{\top }\times {\mathrm{emb}\left(P\right)}_t\right)}\right)-& \left(13\right)\end{array}$$\frac{1}{❘T^{+}❘}\stackrel{❘T^{+}❘}{\sum _j}\log \left(\frac{\exp \left(\mathrm{emb}{\left(P\right)}_k^{\top }\times {\mathrm{ext}\left(T\right)}_j\right)}{\underset{i}{\sum ^N}\exp \left({\mathrm{emb}\left(P\right)}_k^{\top }\times {\mathrm{ext}\left(T\right)}_t\right)}\right)$

where P⁺ and I⁺ are the sets of positive prompts and image tokens corresponding to the selected enc(I)_i and emb(P)_k, respectively.

[0064]Objectness Losses. To model the track's temporal changes, Applicant's network learns from training samples that capture both appearance and motion generated by two adjacent frames:

$\begin{array}{cc}{L}_{I❘T}=-\stackrel{N}{\sum _j}\log \left(\frac{\exp \left(\mathrm{ext}{\left(T\right)}_j^{\top }\times {\mathrm{enc}\left(I\right)}_i\right)}{\underset{i}{\sum ^K}\exp \left({\mathrm{ext}\left(T\right)}_j^{\top }\times {\mathrm{enc}\left(I\right)}_t\right)}\right),\mathrm{and}{L}_{\mathrm{GIoU}}=\stackrel{N}{\sum _j}{\ell }_{\mathrm{GIoU}}\left({\mathrm{tr}}_j,{\mathrm{obj}}_i\right)& \left(14\right)\end{array}$

[0065]Experimental Results: Implementation Details: Experimental Scenarios. Applicant creates three types of prompt: category name nm, category synonyms syn, and category definition def. One tracklet captions cap scenario is constructed by Applicant's detailed annotations and one more objects retrieval retr scenario is given in Applicant's custom request prompts. The dataset contains 833 classes, each has a name and a corresponding set of synonyms that are different names for the same category, such as [man, woman, human, pedestrian, boy, girl, child] for person. Additionally, each category is described by a category definition sentence. This definition makes the model deal with the variations in the text prompts. Applicant joins the names, synonyms, definitions, or captions and filter duplicates to construct the prompt. Trained models use as the same type as testing. Applicant annotated the raw tracking data of the best-performant tracker (i.e., BoT-SORT at 80.5% MOTA and 80.2% IDF1) at the time Applicant constructed experiments and used it as the sub-optimal ground truth of MOT17 and MOT20 (parts (2, 4) in Table 3). That is also the raw data used to evaluate all Applicant's ablation studies.

[0066]Datasets and Metrics. RefCOCO+ and Flickr30 k serve as pre-trained datasets for acquiring a vocabulary of visual-textual concepts. The ext(⋅) operation is not involved in this training step. After obtaining a pre-trained model from RefCOCO+ and Flickr30 k, Applicant trained and evaluated the model for the proposed Type-to-Track task on all five scenarios on the GroOT dataset and the first-three scenarios for MOT20. The tracking performance is reported in class-agnostic metrics CA-MOTA, CA-IDF1, and CA-HOTA as outlined above, and mAP50 as defined previously defined.

[0067]Tokens Production. emb(⋅) utilizes ROBERTa to convert the text input into a sequence of numerical tokens. The tokens are fed into the ROBERTa-base model for text encoding using a 12-layer transformer network with 768 hidden units and 12 self-attention heads per layer. enc(⋅) is implemented using a ResNet-101 as the backbone to extract visual features from the input image. The output of the ResNet is processed by a Deformable DETR encoder to generate visual tokens. For each dimension, Applicant uses sine and cosine functions with different frequencies as positional encodings. A feature resizer combining a list of (Linear, LayerNorm, Dropout) is used to map to size D=512 for all token producers.

[0068]Ablation Study: Comparisons in Different Scenarios. Table 4 shows comparisons in the performance of different prompt inputs. For MOT17 and MOT20, the category name is ‘person’, while category definition is ‘a human being’. Since the prompt by category definition is short, it does not differ much from the nm setting. However, the syn setting shuffles between some words, resulting in a slight decrease in CA-MOTA and CA-IDF1. The cap setting results in prompts that contain more diverse and complex vocabulary, and more context-specific information. It is more difficult for the model to accurately localize the objects and identify their identity within the image, as it needs to take into account a wider range of linguistic cues, resulting in a decrease in performance compared to def (59.5% CA-MOTA and 54.8% CA-IDF1 vs 67.3% CA-MOTA and 72.4% CA-IDF1 on MOT17).

[0069]For TAO, the def setting has a significant number of variations and many tenuous connections in the scene context, for example, ‘an aircraft that has a fixed wing and is powered by propellers or jets’ for the airplane category. Therefore, it results in a decrease in performance (16.8% CA-MOTA and 27.7% CA-IDF1) compared to cap (20.7% CA-MOTA and 32.0% CA-IDF1), because the cap setting is more specific on the object level than category level. The best performant setting is nm (27.3% CA-MOTA and 37.2% CA-IDF1), where names are combined.

[0070]Simplified Attention Representations. Table 4 also presents the effectiveness of different attention representations of the full tensor T (denoted by χ) and the simplified correlation (denoted by ✓). The performance is reported with frame per second (FPS), which is self-measured on one GPU NVIDIA RTX 3060 12 GB. Overall, the performance of simplified correlation is witnessed with a superior speed of up to 2×(7.8 FPS vs 3.4 FPS of cap on MOT17 and 11.5 FPS vs 7.6 FPS of retr on TAO), resulting in and a slight increase in accuracy due to attention stability and precision gain.

[0071]Comparisons with a Baseline Design. Due to the new proposed topic, no current work has the same scope or directly solves Applicant's problem. Therefore, Applicant compares the proposed MENDER against a two-stage baseline tracker in Table 5. Applicant uses current SOTA methods to develop this approach, i.e., MDETR for the grounded detector, while TrackFormer for the object tracker. It is worth noting that the MENDER relies on direct regression to locate and track the object of interest, without the need for an explicit grounded object detection stage. Table 5 shows the proposed MENDER outperforms the baseline on both CA-MOTA and CA-IDF1 metrics in all four settings category synonyms, category definition, tracklet captions and object retrieval (25.7% vs. 21.3%, 16.8% vs. 14.6%, 20.7% vs. 15.3% and 32.9% vs. 25.7% CA-MOTA on TAO), while can maintain up to 4× run-time speed (10.3 FPS vs 2.2 FPS). The results indicate that training a single-stage network enhances efficiency and reduces errors by avoiding separate feature extractions for both detection and tracking steps.

[0072]Comparisons with State-of-the-Art Approaches. The category name nm setting is also the official MOT benchmark. Table 6 is the comparison of the result on the category name setting on the official leaderboard of MOT17, compared with other state-of-the-art approaches, including ByteTrack and TrackFormer. Note that the proposed MENDER is one of the first attempts at the Grounded MOT task, not to achieve the top rankings on the general MOT leaderboard. In contrast, other SOTA approaches benefit from the efficient single-category design in their separate object detectors, while the single-stage design is agnostic to the category and for flexible textual input. Compared to TrackFormer, the proposed MENDER only demonstrates a marginal decrease in identity assignment (67.1% vs 68.0% CA-IDF1). The decrease in the CA-MOTA stems from the detector's design which integrates flexible input.

[0073]Conclusion. Applicant has presented a novel problem of Type-to-Track, which aims to track objects using natural language descriptions instead of bounding boxes or categories, and a large-scale dataset to advance this task. The proposed MENDER model reduces the computational complexity of third-order correlations by designing an efficient attention method that scales quadratically with respect to the input sizes. The experiments on three datasets and five scenarios demonstrate that the model achieves state-of-the-art accuracy and speed for class-agnostic tracking.

[0074]The Type-to-Track problem and the proposed MENDER model have the potential to impact various fields, such as surveillance and robotics, where recognizing object interactions is a crucial task. By reformulating the problem with text support, the proposed methodology can improve the intuitiveness and responsiveness of tracking, making it more practical for video input support in large-language models and real-world applications.

[0075]Although various embodiments of the present disclosure have been illustrated in the accompanying Drawings and described in the foregoing Detailed Description, it will be understood that the present disclosure is not limited to the embodiments disclosed herein, but is capable of numerous rearrangements, modifications, and substitutions without departing from the spirit of the disclosure as set forth herein.

[0076]The term “substantially” is defined as largely but not necessarily wholly what is specified, as understood by a person of ordinary skill in the art. In any disclosed embodiment, the terms “substantially”, “approximately”, “generally”, and “about” may be substituted with “within [a percentage] of” what is specified, where the percentage includes 0.1, 1, 5, and 10 percent.

[0077]The foregoing outlines features of several embodiments so that those of ordinary skill in the art may better understand the aspects of the disclosure. Those of ordinary skill in the art should appreciate that they may readily use the disclosure as a basis for designing or modifying other processes and structures for carrying out the same purposes and/or achieving the same advantages of the embodiments introduced herein. Those of ordinary skill in the art should also realize that such equivalent constructions do not depart from the spirit and scope of the disclosure, and that they may make various changes, substitutions, and alterations herein without departing from the spirit and scope of the disclosure. The scope of the invention should be determined only by the language of the claims that follow. The term “comprising” within the claims is intended to mean “including at least” such that the recited listing of elements in a claim are an open group. The terms “a”, “an”, and other singular terms are intended to include the plural forms thereof unless specifically excluded.

[0078]Conditional language used herein, such as, among others, “can”, “might”, “may”, “e.g.”, and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements, and/or states. Thus, such conditional language is not generally intended to imply that features, elements, and/or states are in any way required for one or more embodiments.

[0079]While the above detailed description has shown, described, and pointed out novel features as applied to various embodiments, it will be understood that various omissions, substitutions, and changes in the form and details of the embodiments illustrated can be made without departing from the spirit of the disclosure. As will be recognized, the various embodiments described herein can be embodied within a form that does not provide all of the features and benefits set forth herein, as some features can be used or practiced separately from others. The scope of protection is defined by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.