FANUC CORPORATION

A method for identifying inaccurately depicted boxes in an image, such as miss detected boxes and partially detected boxes. The method obtains a 2D RGB image of the boxes and a 2D depth map image of the boxes using a 3D camera, where pixels in the depth map image are assigned a value identifying the distance from the camera to the boxes. The method generates a segmentation image of the boxes using a neural network by performing an image segmentation process that extracts features from the RGB image and segments the boxes by assigning a label to pixels in the RGB image so that each box in the segmentation image has the same label and different boxes in the segmentation image have different labels. The method analyzes the segmentation image to determine if the image segmentation process has failed to accurately segment the boxes in the segmentation image.

A method for dual hand detection in robot teaching from human demonstration. A camera image of the demonstrator's hands and workpieces is provided to a first neural network which determines the identity of the left and right hand of the human demonstrator from the image, and also provides cropped sub-images of the identified hands. The first neural network is trained using images in which the left and right hands are pre-identified. The cropped sub-images are then provided to a second neural network which detects the pose of both the left and right hand from the images, where the sub-image for the left hand is horizontally flipped before and after the hand pose detection if second neural network is trained with right hand images. The hand pose data is converted to robot gripper pose data and used for teaching a robot to perform an operation through human demonstration.

A system and method for identifying a box to be picked up by a robot from a stack of boxes. The method includes obtaining a 2D red-green-blue (RGB) color image of the boxes and a 2D depth map image of the boxes using a 3D camera. The method employs an image segmentation process that uses a simplified mask R-CNN executable by a central processing unit (CPU) to predict which pixels in the RGB image are associated with each box, where the pixels associated with each box are assigned a unique label that combine to define a mask for the box. The method then identifies a location for picking up the box using the segmentation image.

A system and method for training a neural network. The method includes modelling a plurality of different sized objects to generate virtual images of the objects using computer graphics software and generating a placement virtual image by randomly and sequentially selecting the modelled objects and placing the selected modelled objects within a predetermined boundary in a predetermined pattern using the software. The method also includes rendering a virtual image of the placement virtual image based on predetermined data and information using the computer graphics software and generating an annotated virtual image by independently labeling the objects in the rendered virtual image using the software. The method repeats generating a placement virtual image, rendering a virtual image and generating an annotated virtual for a plurality of randomly and sequentially selected modelled objects, and then trains the neural network using the plurality of rendered virtual images and the annotated virtual images.

A control device includes a processing unit for processing a first image captured by a first camera and a second image captured by a second camera. The processing unit performs control using block matching. A first determination value on a block matching score and a second determination value greater than the first determination value are determined in advance. The processing unit generates a first distance image using the first determination value and a second distance image using the second determination value. The processing unit adopts distance information for a pixel contained in the first distance image for a region corresponding to the contour of a workpiece and distance information for a pixel contained in the second distance image for a region other than the region corresponding to the contour to generate a synthesized distance image.

An image processing apparatus includes: a model pattern storage unit that stores a model pattern composed of a plurality of model feature points; an image data acquisition unit that acquires a plurality of images obtained through capturing an object to be detected; an object detection unit that detects the object to be detected from the images using the model pattern; a model pattern transformation unit that transforms a position and posture such that the model pattern is superimposed on an image of the object to be detected; a corresponding point acquisition unit that acquires a corresponding point on image data corresponding to each of the model feature points; a corresponding point set selection unit that selects a set of corresponding points on the plurality of images; and a three-dimensional position calculation unit that calculates a three-dimensional position of the image of the object to be detected.

The present invention facilitates detection of an object imaged by a three-dimensional sensor. The image processing device includes: a distance calculating unit that, on the basis of three-dimensional data acquired by the three-dimensional sensor, calculates a distance between each of points in the three-dimensional data and a reference plane; a distance image creating unit that creates a distance image having pixel values constituted by values each calculated on the basis of the distance calculated by the distance calculating unit; and an image processing unit that carries out image processing on the distance image. The reference plane may be a bearing surface bearing the object or a surface parallel with the bearing surface. The reference plane may also be a surface of the object. The image processing device may include the three-dimensional sensor.

Provided is an image processing device with which it is possible to automate the setting of a detection parameter that is suitable for obtaining a desired detection result. An image processing device is provided with: an object detection unit for detecting the image of an object from input image data using a detection parameter; a detection rate calculation unit for comparing the result of detection by the object detection unit with information that represents a desired detection result so as to calculate at least one of a non-detection rate and a false detection rate in object detection by the objection detection unit; an objective function value calculation unit for calculating the value of an objective function, the input variable of which is at least one of the non-detection rate and the false detection rate; and a detection parameter search unit for performing a search of the detection parameter by changing the value of the detection parameter and repeating object detection, calculation of at least one of the non-detection rate and the false detection rate, and calculation of the value of the objective function until the value of the objective function satisfies a prescribed condition or the number of times a search of the detection parameter is performed reaches a prescribed count.

The present invention addresses the problem of providing an image processing system that can accurately and efficiently detect a target object even when the positional relationship between an image capturing device and the target object differs between the time of teaching and the time of detection. An image processing system 1 includes: a control unit 52 that obtains, on the basis of position information of a robot 2 for specifying the position of a visual sensor 4 in a robot coordinate system and on the basis of position information showing the position of a target object W in an image coordinate system, the positional relationship between the visual sensor 4 and the target object W; and a storage unit 51 that stores, on the basis of a model pattern consisting of feature points extracted from a teaching image and on the basis of the positional relationship between the visual sensor 4 and the target object W when capturing the teaching image, the model pattern in the form of three-dimensional position information. The control unit 52 performs detection processing, in which the target object W is detected from a detection image on the basis of a result obtained by matching the model pattern with the feature points extracted from the detection image including the target object W.

Provided is an object detection device which can detect with high accuracy a symmetrical target object represented in an image. The object detection device includes a memory which stores a model pattern representing a plurality of predetermined features in different positions to each other on a target object when the target object is viewed from a predetermined direction, a feature extraction unit which extracts a plurality of the predetermined features from an image in which the target object is represented, and a collation unit which calculates a degree of coincidence representing a degree of matching between the plurality of the predetermined features of the model pattern and a plurality of the predetermined features extracted from a region corresponding to the model pattern in the image while changing at least one of a relative position or angle, or a relative direction and relative size of the model pattern with respect to the image, and which judges that the target object is represented in the region of the image corresponding to the model pattern when the degree of coincidence is equal to or greater than a predetermined threshold, wherein the predetermined features stored in the memory include a feature of interest which can be used for detecting a position in a specific direction of the target object in the image or which can be used for detecting an angle in a rotational direction centered about a predetermined point of the target object in the image, and the collation unit increases the contribution in the calculation of the degree of coincidence when the feature of interest of the model pattern and a predetermined feature in the image match as compared to the case in which a predetermined feature of the model pattern other than the feature of interest and the predetermined feature in the image match to calculate the degree of coincidence.