Five AI Limited

A computer-implemented method of planning ego actions for a mobile robot in the presence of at least one agent, comprising: searching for an optimal ego action in multiple search steps, each comprising: selecting an ego action from a set of possible ego actions, selecting an agent behaviour from a set of possible agent behaviours, running a simulation based on the selected ego action and agent behaviour, determining a possible outcome, and assigning a reward to the selected ego action, based on a reward metric, wherein selection of the ego action in later search steps is biased towards higher reward ego action(s) but selection of the agent behaviour in later search steps is biased towards riskier agent behaviour(s), a risky agent behaviour being, according to earlier search steps, more likely to result in a lower reward outcome and choosing an ego action based on the rewards computed in the search steps.

A computer system for rendering a graphical user interface for visualising runs of a driving scenario in which an ego agent navigates a road layout, comprising an input configured to receive a map of the road layout and run data comprising a sequence of timestamped ego agent states and a time-varying numerical score quantifying the performance of the ego agent with respect a set of run evaluation rules; and a rendering component configured to cause a graphical user interface to display, for each rule: a plot of the time-varying numerical score, and a marker denoting a selected time index of the plot, the marker movable along the time axis to change the selected time index; and a scenario visualization comprising a visualization of the run at the selected time index, whereby moving the marker along the time axis causes the scenario visualisation to update as the time index is changed.

A computer-implemented method of modelling a perception system for perceiving objects captured in sensor data comprises: receiving a plurality of training examples, each comprising a ground truth scene for a set of sensor data and a corresponding perceived scene obtained by applying the perception system to the set of sensor data; fitting to the training examples noise model parameters, encoding a noise distribution over perceived scenes given a misdetection scene, and misdetection model parameters, encoding a misdetection distribution over misdetection scenes given a ground truth scene; computing a perception distribution over perceived scenes for a given ground truth scene by marginalizing the product of noise and misdetection distributions over multiple misdetection scenes, wherein individual objects in the ground truth scene are not associated with individual objects in the perceived scenes; fitting the noise and misdetection model parameters to match the perception distribution to the perceived scene for each training example.

A computer-implemented method of creating one or more annotated perception inputs, the method comprising, in an annotation computer system: receiving a plurality of captured frames, each frame comprising a set of 3D structure points, in which at least a portion of a common structure component is captured; computing a reference position within at least one reference frame of the plurality of frames; generating a 3D model for the common structure component by selectively extracting 3D structure points of the reference frame based on the reference position within that frame; determining an aligned model position for the 3D model within a target frame of the plurality of frames based on an automatic alignment of the 3D model with the common structure component in the target frame; and storing annotation data of the aligned model position in computer storage, in association with at least one perception input of the target frame for annotating the common structure component therein.

A method of predicting trajectories for agents of a scenario, the method comprising, for each agent generating an agent feature vector based on one or more observed past states of the agent, computing a set of pairwise feature vectors, each computed as a combination of the agent feature vector for that agent with a respective agent feature vector generated for each other agent of the scenario, processing the pairwise feature vectors as independent inputs to one or more interaction layers of a trajectory prediction neural network to generate a pairwise output for each pairwise feature vector, aggregating the pairwise outputs over the other agents of the scenario to generate an interaction-based feature representation for each agent, processing the interaction-based feature representation in one or more prediction layers of the trajectory prediction neural network, and generating, based on the output of the one or more prediction layers, at least one predicted trajectory for each agent.

A method for testing performance of a stack for planning ego vehicle trajectories in real or simulated driving scenarios applying driving rules to the scenario ground truth for evaluating the performance of the stack in the scenario, and providing output indicating whether each driving rule has been complied with; wherein the driving rules include at least one ODD-based response rule, wherein applying the ODD-based response rule includes processing the scenario ground truth over multiple time steps, to determine whether the scenario is within the defined ODD at each time step, and thereby detecting a change in the scenario that takes the scenario outside the defined ODD, and processing the internal state data, to determine whether a state change occurred within the stack, within a time interval, the output for the at least one ODD-based response rule indicating whether the state change occurred within the time interval.

A computer system configured to run queries on a static road layout, the computer system comprising: computer storage configured to store the static road layout, the static road layout comprising a section of road having a set of multiple road attributes, each road attribute described throughout the section of road by a describing function that exhibits a change in form at one or more change points along the section of road, the change points of a first of the road attributes exhibiting longitudinal misalignment with respect to the change points of a second of the road attributes; a road partitioning component configured to process the static road layout, and thereby partition the section of road into a sequence of road parts, each road part defined by a longitudinal coordinate interval, in which the describing function of every one of the road attributes has a form that is fixed throughout; a road indexing component configured to generate a road partition index having an entry for each road part, the entry indicating the form of the describing function of each road attribute as fixed throughout the longitudinal coordinate interval of that road part; and a scenario query engine configured to receive a part query, locate the entry in the road partition index for one of the road parts based on the part query, evaluate the describing function of at least one of the road attributes within the road part, and generate a part query response based on the evaluation of the describing function.

A computer system for generating a scenario to be run in a simulation environment for testing the behaviour of an autonomous vehicle, the computer system comprising: a rendering component configured to: generate display data for causing a display to render a graphical user interface comprising an image of a driving environment and one or more agents within the driving environment; a parameter generator configured to generate in memory a user-defined parameter set responsive to user input defining the parameter set; and an expression manager configured to store in memory a user-defined expression set, responsive to user input defining the expression set, wherein each expression of the expression set is a user-defined function of one or more parameters of the parameter set; and a scenario generator configured to record the scenario in a scenario database; wherein the graphical user interface is configured to provide multiple agent fields for controlling the behaviour of the one or more agents when the scenario is run in a simulation environment, wherein each agent field is modifiable to associate therewith either a parameter of the user-defined parameter set or an expression of the user-defined expression set; and wherein the recorded scenario comprises the driving environment, the one or more agents, the user-defined parameter set, the user-defined expression set, and any user-defined associations between (i) the multiple agent fields and the user-defined parameter set and (ii) the multiple agent fields and the user-defined expression set, wherein each parameter associated with an agent field is controllable to directly modify an agent behaviour, and each parameter that is included in expression associated with an agent field is controllable to indirectly modify an agent behaviour.

The disclosure provides systems and methods for identifying salient test runs involving an autonomous vehicle system. A processor receives sets of run data, each set representative of a driving scenario. For each set, an output set is generated, the output set comprising time-indexed events generated in response to a detected behaviour of at least one challenger agent, and a sequence of decision indicators indicating whether a driving action by an ego agent would be permissible. A data retrieval component is coupled to a results database and retrieves output sets based on the time-indexed events and the sequence of decision indicators. The processor generates the sequence of decision indicators by generating a planned trajectory of the ego agent, and determining whether the predefined driving action by the ego agent would be permissible.

Herein, a “perception statistical performance model” (PSPM) for modeling a perception slice of a runtime stack for an autonomous vehicle or other robotic system may be used e.g. for safety/performance testing. A PSPM is configured to: receive a computed perception ground truth t; determine from the perception ground truth t, based on a set of learned parameters, a probabilistic perception uncertainty distribution of the form p(e|t), p(e|t,c), in which p(e|t,c) denotes the probability of the perception slice computing a particular perception output e given the computed perception ground truth t and the one or more confounders c, and the probabilistic perception uncertainty distribution is defined over a range of possible perception outputs, the parameters learned from a set of actual perception outputs generated using the perception slice to be modeled, wherein each confounder is a variable of the PSPM whose value characterized a physical condition on which p(e|t,c) depends.