Boston Dynamics, Inc.

Systems and methods are described for combining sensor data obtained by a mobile robot. A system can obtain first sensor data from one or more first sensors of a robot. The system can determine a distance between the robot and at least a portion of the environment based on the first sensor data. For example, the distance may be a depth from a depth map. The system can obtain second sensor data from one or more second sensors of the robot. The system can combine a first portion of the second sensor data and a second portion of the second sensor data based on the distance. For example, the system can use the distance to determine a seam for combination of the first image and the second image. The system can instruct output of a user interface based on the combination.

Systems and methods are described for mapping and traversal of a set of stairs. A system can obtain a map of an environment. The system can determine that the environment includes a set of stairs based on the map. For example, the map may indicate the presence of the set of stairs within the environment. For mapping the set of stairs, the system can instruct performance of a stair mapping maneuver based on the determination that the environment includes the set of stairs. Based on the performance of the stair mapping maneuver, the system can map the set of stairs. The system can instruct traversal of the set of stairs by a robot based on the mapping of the set of stairs.

Systems and methods are described for provision of alerts. A system can obtain sensor data associated with an environment. The system can obtain an input indicating one or more filter parameters, one or more requests, and/or one or more alert parameters. For example, the input may indicate a request, a region of the environment, a region of sensor data, etc. The system can filter the sensor data based on the one or more filter parameters to obtain a filtered portion of the sensor data. The system may generate a prompt for a machine learning model that includes the filtered portion of the sensor data, the one or more requests, and the one or more alert parameters. The system can provide the prompt to a computing system. The system can obtain a output from the computing system and can provide an alert based on the output.

Systems and methods for semantic robot hazard avoidance with multi-modal prompting are provided. In one aspect, a method includes receiving a user input indicative of one or more hazards in an environment of the robot and image data indicative of the one or more hazards in the environment. The method also includes generating one or more segments of the image data. Each of the one or more segments corresponds to at least one of the one or more hazards indicated by the user input. The method further includes identifying a semantic label for each of the one or more segments, generating a hazard map including a location of each of the one or more segments and the corresponding semantic label, and navigating the robot through the environment based at least in part on the hazard map.

A computer-implemented method executed by data processing hardware of a robot causes the data processing hardware to perform operations. The operations include receiving a sensor pointing command that commands the robot to use a sensor to capture sensor data of a location in an environment of the robot. The sensor is disposed on the robot. The operations include determining, based on an orientation of the sensor relative to the location, a direction for pointing the sensor toward the location, and an alignment pose of the robot to cause the sensor to point in the direction toward the location. The operations include commanding the robot to move from a current pose to the alignment pose. After the robot moves to the alignment pose and the sensor is pointing in the direction toward the location, the operations include commanding the sensor to capture the sensor data of the location in the environment.

A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations. The operations include detecting a candidate support surface at an elevation less than a current surface supporting a legged robot. A determination is made on whether the candidate support surface includes an area of missing terrain data within a portion of an environment surrounding the legged robot, where the area is large enough to receive a touchdown placement for a leg of the legged robot. If missing terrain data is determined, at least a portion of the area of missing terrain data is classified as a no-step region of the candidate support surface. The no-step region indicates a region where the legged robot should avoid touching down a leg of the legged robot.

A method includes receiving, while a robot traverses a building environment, sensor data captured by one or more sensors of the robot. The method includes receiving a building information model (BIM) for the environment that includes semantic information identifying one or more permanent objects within the environment. The method includes generating a plurality of localization candidates for a localization map of the environment. Each localization candidate corresponds to a feature of the environment identified by the sensor data and represents a potential localization reference point. The localization map is configured to localize the robot within the environment when the robot moves throughout the environment. For each localization candidate, the method includes determining whether the respective feature corresponding to the respective localization candidate is a permanent object in the environment and generating the respective localization candidate as a localization reference point in the localization map for the robot.

A computer-implemented method, executed by data processing hardware of a robot, includes receiving a three-dimensional point cloud of sensor data for a space within an environment about the robot. The method includes receiving a selection input indicating a user-selection of a target object represented in an image corresponding to the space. The target object is for grasping by an end-effector of a robotic manipulator of the robot. The method includes generating a grasp region for the end-effector of the robotic manipulator by projecting a plurality of rays from the selected target object of the image onto the three-dimensional point cloud of sensor data. The method includes determining a grasp geometry for the robotic manipulator to grasp the target object within the grasp region. The method includes instructing the end-effector of the robotic manipulator to grasp the target object within the grasp region based on the grasp geometry.

A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations. The operations include detecting a candidate support surface at an elevation less than a current surface supporting a legged robot. A determination is made on whether the candidate support surface includes an area of missing terrain data within a portion of an environment surrounding the legged robot, where the area is large enough to receive a touchdown placement for a leg of the legged robot. If missing terrain data is determined, at least a portion of the area of missing terrain data is classified as a no-step region of the candidate support surface. The no-step region indicates a region where the legged robot should avoid touching down a leg of the legged robot.

A computer-implemented method executed by data processing hardware of a robot causes the data processing hardware to perform operations. The operations include receiving a sensor pointing command that commands the robot to use a sensor to capture sensor data of a location in an environment of the robot. The sensor is disposed on the robot. The operations include determining, based on an orientation of the sensor relative to the location, a direction for pointing the sensor toward the location, and an alignment pose of the robot to cause the sensor to point in the direction toward the location. The operations include commanding the robot to move from a current pose to the alignment pose. After the robot moves to the alignment pose and the sensor is pointing in the direction toward the location, the operations include commanding the sensor to capture the sensor data of the location in the environment.