DETERMINING BALL IMPACT LOCATION ON SURFACE OF STRIKING OBJECT

TECHNICAL FIELD

[0001]The disclosed embodiments are related to determining an impact location on a surface of a striking object striking a ball.

BACKGROUND

[0002]Striking a golf ball in the center of a clubface of a golf club (commonly referred to as the “sweet spot”) will generally yield good results in terms of overall performance. For example, when the golf ball is struck at the “sweet spot,” the golf ball will typically travel maximum distance along an optimal trajectory with better control. The “sweet spot” provides the most efficient transfer of energy from the golf club to the golf ball, resulting in a more consistent and accurate shot.

[0003]One method to identify the impact location on a clubface is to apply a sticker to the clubface. When the clubface strikes the golf ball, the sticker changes color, thereby indicating the impact location on the clubface. This feedback allows the golfer to make necessary adjustments to their swing technique or club settings to improve their overall performance. This method, however, may leave adhesive residue on the clubface which can be burdensome to remove.

[0004]Another method computes a geometric center of the clubface, the ball's position before impact, the time of impact, the position of the clubface, the angle of attack on the ball, and the dynamic loft to identify the impact location. The time of impact is when the clubface comes into contact with the golf ball. Since the time of impact is calculated using interpolation between time points, a ball impact position determined based on time of impact will typically be inaccurate.

SUMMARY

[0005]Embodiments are disclosed for determining a ball impact location on a surface of a striking object.

[0006]In some embodiments, a method comprises: capturing a series of images including at least three images of a striking object prior to impact with a stationary ball; determining, based on center coordinates of surfaces of the striking object captured in the at least three images, a circumference of a circle; shifting the ball along the circumference of the circle to a virtual position such that the ball at the virtual position is in contact with a first surface of the striking object; and determining a first impact location on the first surface of the striking object in contact with the shifted ball at the virtual position.

[0007]In some embodiments, the method further comprises determining a second impact location on a second surface of the striking object in contact with the stationary ball based on the first impact location to determine a flight path of the ball, wherein the first impact location and the second impact location are the same location relative to the surface of the club head, but their locations in 3D space are different with respect to a fixed point.

[0008]In some implementations, determining the circumference of the circle comprises using a circle fitting algorithm to fit the center coordinates of the at least one surface of the at least three images of the striking object to the circumference of the circle.

[0009]In some embodiments, determining the second impact location comprises shifting the determined first impact location by an angle of θ.

[0010]In some embodiments, the method further comprises determining, by a trained neural network, a normal vector and the center coordinates of the at least one surface of the striking object based on the series of images.

[0011]In some embodiments, the trained neural network is an artificial neural network (ANN) comprising a set of nodes organized in layers, the layers including an input layer, at least one hidden layer and an output layer, each layer including at least one node; a set of weights representing connections between the layers; and an activation function that transforms the output of at least one node.

[0012]In some embodiments, training the neural network comprises: extracting features from the series of images; applying a set of weights to the features; applying an activation function to the weighted input features; and predicting a normal vector and center coordinates of the at least one surface of the striking object based on an output of the activation function.

[0013]In some embodiments, the training further comprises training the neural network with training images having different types of striking objects having different loft angles.

[0014]In some embodiments, one of the at least three images includes the ball in a stationary position.

[0015]In some embodiments, the training images further include balls having different circumference.

[0016]Other embodiments are directed to systems, apparatuses and non-transitory, computer-readable storage mediums.

BRIEF DESCRIPTION OF THE DRAWINGS

[0017]FIG. 1 illustrates a process of training a machine learning model, according to one or more embodiments.

[0018]FIG. 2 is a conceptual drawing of a side view of a golf swing, according to one or more embodiments.

[0019]FIG. 3 is a conceptual drawing of another side view of a golf swing with a circle fitting method, according to one or more embodiments.

[0020]FIG. 4 is a conceptual drawing of another side view of a golf swing, according to one or more embodiments.

[0021]FIG. 5 illustrates the impact location on the clubface, according to one or more embodiments.

[0022]FIG. 6 is a flow diagram of a process to train the machine learning model, according to one or more embodiments.

[0023]FIGS. 7A and 7B are flow diagrams of a process to determine an impact vector, according to one or more embodiments.

[0024]FIG. 8 illustrates determining the impact location where a detection unit is placed behind a user, according to one or more embodiments.

[0025]FIG. 9 illustrates determining the impact location where the detection unit of FIG. 8 is placed above the user, according to one or more embodiments.

[0026]FIG. 10 is a block diagram of a system for determining the impact location, according to one or more embodiments.

DETAILED DESCRIPTION

[0027]The disclosed embodiments determine an impact location on a surface of a striking object (e.g., golf clubface) where the striking object impacts the ball. A machine learning model (e.g., a neural network) has been trained to predict a normal vector and center coordinates of a club surface at various positions before the club hits the ball, and center coordinates of the ball at a stationary position such as where the user places the ball before the user strikes the ball. The normal vector and center coordinates of the club surface at various positions before the club hits the ball with the center coordinates of the ball are used to determine from a series of images the impact location on the surface of the striking object at a pre-impact position of the striking object. The pre-impact position of the striking object is then used to determine the impact location on the surface of the striking object.

[0028]In the example shown and the examples that follow, the ball is a golf ball, the striking object is a golf club, and the striking surface is the golf clubface. The disclosed embodiments, however, are also applicable to other types of balls/objects and striking objects and any other balls/objects and striking objects where a ball/object is impacted by a surface of the striking object.

[0029]FIG. 1 illustrates a method of training a machine learning model, according to one or more embodiments. In the example shown, input image 102 includes a user hitting a golf ball. Input image 102 is fed into trained machine learning model 104 which predicts/infers face normal vector 108 and center coordinates 110 of golf club 106. This is achieved by training the machine learning model to extract features from a series of images, applying a set of weights to the features, and applying an activation function based on the weighted input features. From the output of the activation function, the machine learning model predicts the normal vector and center coordinates of a surface of the striking object.

[0030]In some embodiments, the training images may include golf clubs having different loft angle, such as a 7-iron, 6-iron or a driver. In some embodiments, the training images are augmented by at least one of color, contrast, brightness, sharpness, shift or noise level, non-uniform lighting or complex backgrounds. In some embodiments, the training images are randomly augmented. In some embodiments, the training images may include balls having different size, but not limited to golf balls, tennis ball, soccer ball and ping pong ball.

[0031]In some embodiments, trained neural network is an artificial neural network (ANN) comprising a set of nodes organized in layers, the layers including an input layer, at least one hidden layer and an output layer, each layer including at least one node; a set of weights representing connections between the layers; and an activation function that transforms the output of at least one node.

[0032]These parameters such as the center coordinates, face normal vector together with other parameters as described in the following description are then used to determine an impact location on the clubface of golf club 106 as described in further detail below.

[0033]FIG. 2 is a conceptual drawing of a side view of a golf swing, according to one or more embodiments. In some embodiments, an imager (not shown) is used to capture a series of images (e.g., captured by one or more video cameras) of a user swinging the golf club, where the golf club approaches a stationary golf ball 214. The stationary position is referred to in FIG. 2 as P₀. The series of images include three images of golf club 208, 206 and 204 at different positions P₋₁, P₋₂, P₋₃ respectively, which are selected from the series of images during the swing but before the golf club comes into contact with the stationary golf ball 214.

[0034]Note that the image with golf club 208 at position P₋₁ is the only image that contains the stationary golf ball 214 and the other images are golf club 206 and 204 at positions P₋₂, P₋₃, respectively, where golf club 208 in image at position P₋₁ is closest to the stationary golf ball 214, golf club 204 in image at position P₋₃ is the farthest away from the stationary golf ball 214, and golf club 206 in image at position P₋₂ is in between both images at positions P₋₁ and P₋₃. The negative subscripts for positions P₋₁, P₋₂, P₋₃ indicate that the positions are prior to impact of the golf ball. The trained machine learning model 104 is used to determine the face normal vectors 210, 212, 214 and center coordinates 209, 211, 213 of the golf club at positions P₋₁, P₋₂, P₋₃, respectively.

[0035]In some embodiments, the series of images may include more than three images, where one image contains only the ball, and the other images only contain the golf club before the golf club comes into contact with the ball.

[0036]In some embodiments, more than one imager may be employed i.e. two, three, four or more imagers may be used to generate the series of images. In some embodiments, when two or more imagers are used, these imagers may be synchronized with each other and the two or more imagers are placed in parallel to one another towards the object of interest, in this instance, the ball. In some embodiments, the imager may be a stereo camera having two or more lenses.

[0037]FIG. 3 is a conceptual drawing of another side view of a golf swing illustrating a circle fitting method, according to one or more embodiments. Three golf club center coordinates 209, 211, and 213 of the golf club at positions P₋₁, P₋₂, P₋₃ are used to fit a circle. If all three golf club center coordinates of the golf club at positions P₋₁, P₋₂, P₋₃, 209, 211, 213 fall within a circumference of the circle (shown by dot dash line 304) a circle will be formed. The circle may be used to determine the center coordinate of the circle which is also referred as rotation point 301 and may be used to determine the normal vector 302 at the rotation point 301. This is achieved by using the three golf club center coordinate parameters and inputting these parameters into a general equation for a circle (x−x_c)²+(y−y_c)²=r². Similarly, the normal vector 302 (N_x, N_y, N_z) is also determined by a general vector equation N_x(x−x_c)²+N_y(y−y_c)²+N_z(z−z_c)²=0.

[0038]In some embodiments, linear methods such as Full Least Squares Method, Average of Intersections Method, Reduced Least Square Method, and Modified Least Squares Method may be used for the circle fitting method. In some embodiments, non-linear methods such as machine learning may be used for the circle fitting method.

[0039]FIG. 4 is a conceptual drawing of another side view of a golf swing, according to one or more embodiments. Using the circumference of the circle 304 determined in FIG. 3 and using the center coordinates 403 of the stationary ball 214, the stationary ball 214 is moved clockwise 411 along the circumference of the circle 304 relative to the center coordinates of the rotation point 301 (Cx, Cy, Cz), and with respect to the swing normal vector 302 (Nx, Ny, Nz) towards the golf club 208. As the stationary ball 214 moves, its outer circumference touches a surface of the golf club 208, where the touched surface is the golf clubface to a position, the position is referred as a virtual position and the ball at the virtual position is referred to as a virtual ball 414 having a center coordinate 401. At the virtual position, outer circumference of the virtual ball 414 touches the surface of the golf club 208 at a contact point P′c 409 with respect to the club surface normal vector N′c 402. The normal vector N′c 402 dictates the orientation of the club face. For example, if the normal vector is pointing towards the direction on the right, the club face will also point towards the direction on the right.

[0040]Having determined the contact point P′c 409, the golf club 208 is shifted towards stationary ball 214 about an angle θ relative to the rotation point 301 (Cx, Cy, Cz) and with respect to the swing normal vector 302 (Nx, Ny, Nz). This shifting positions a club surface of the golf club 208 to touch an outer circumference of the stationary ball 214 at position P₀. In this instance, the golf club is referred to as virtual golf club 210. At the position P₀, a surface of the virtual golf club 210 touches an outer circumference of the stationary ball 214 with respect to the normal vector Nc 412 of club face normal vector and this touch point is referred as contact point Pc 405 or impact location. The impact location 405 at position P₀ and the normal vector Nc 412 at the virtual position is described by equations [1] to [3] following below.

[0041]In some embodiment, the stationary ball 214 may be moved clockwise along the circumference of the circle 304 relative to the center coordinates of the rotation point 301 (Cx, Cy, Cz), and with respect to the swing normal vector 302 (Nx, Ny, Nz) towards the golf club 208 or 204.

[0042]Assume the Cartesian coordinates of P′c is x, y, z. As P′c is on the golf club face, then

$\begin{array}{cc}{N}_x\left(x-P_x\right)+N_y\left(y-P_y\right)+N_z\left(z-P_z\right)=0& \left[1\right]\end{array}$

where N_x, N_y, N_z, are the normal vector of the clubface and P_x, P_y, P_z, are the center point of the club face in Cartesian coordinates, and both are obtained by a machine learning model (e.g., neural network).

[0043]Let the virtual ball 414 in Cartesian coordinates be x₀, y₀, z₀, r₀, where x₀ is the x-coordinate of the virtual ball, y₀ is the y-coordinate of the virtual ball, z₀ is the z-coordinate of the virtual ball, and r₀ is the radius of the virtual ball. Since position P′c is touching the virtual ball 414 then

$\{\begin{array}{c}{x}_0=x-N_{x}r0\\ y_0=y-N_{y}r0\\ z_0=z-N_{z}r0\end{array}$

[0044]The distance of the virtual ball 414 to the center point of the swing is determined by center coordinates 301 (C_x, C_y, C_z) and the swing normal vector 302 (N_sx, N_sy, N_sz) is l, where l is the distance of the stationary ball to the center point of the swing determined by the center coordinates 301 (C_x, C_y, C_z) and the swing normal vector 302 (N_sx, N_sy, N_sz), giving:

$\begin{array}{cc}{\left(x_0-C_x\right)}^2+{\left(y_0-C_y\right)}^2+{\left(z_0-C_z\right)}^2=l^2.& \left[2\right]\end{array}$

[0045]And x₀, y₀, z₀, are on the plane formed by stationary ball x′₀, y′₀, z′o with swing normal vector (N_sx, N_sy, N_sz) give:

$\begin{array}{cc}{N}_x\left(x_0-x_0^{\prime }\right)+N_y\left(y_0-y_0^{\prime }\right)+N_z\left(z_0-z_0^{\prime }\right)=0.& \left[3\right]\end{array}$

[0046]Combining equations [1], [2], and [3], x, y, z are the coordinates of position P′c, normal vector N′_e is the club face normal vector, θ theta is the angle between position x₀, y₀, z₀, and x′₀, y′₀, z′₀, and the final impact location is determined by rotating x, y, z by angle θ theta with respect to the normal vector N′_c. Other parameters such as the ball's trajectory or flight path after being struck by the club, may be determined based on the final impact location and the normal vector of the club at the final impact location. Note that the final impact location and previous impact location are the same location relative to the surface of the club head, but their locations in 3D space are different with respect to a fixed point.

[0047]FIG. 5 illustrates the impact location on the club face 208 according to one or more embodiments. After determining the impact location as described in FIGS. 1-4, the impact location 502 is shown to be on the clubface 208.

[0048]FIG. 6 is a flow diagram of process 600 to train a machine learning model, according to one or more embodiments. Process 600 begins by feeding a series of training images into a machine learning model (e.g., a neural network or a trained neural network) 602. The training images can be synthetically generated images or real images. In some embodiments, the images are indoor or outdoor images of a user swinging striking objects at a ball. In the case of golf, the images are images of golfer swinging a golf club at a golf ball. In some embodiments, the training images may include different types of striking objects having different loft angles. In some embodiments, the training images may be augmented by at least one of color, contrast, brightness, sharpness, shift or noise level, non-uniform lighting or complex backgrounds. In some embodiments, at least one of the training images may be randomly augmented. In some embodiments, the training images may include different types of striking objects having different loft angles.

[0049]Process 604 continues by identifying the striking object (e.g., a golf club) in a series of images using an image-based object recognition network, such as, e.g., YOLO or R-CNN (604).

[0050]After the machine learning model has identified the striking object in the series of images, process 600 continues with the machine learning model determining a normal vector and center coordinates of the face of the striking object (608). In some embodiments, following the training, training data is stored as a dataset in the system. Each of the foregoing steps was previously described in reference to FIGS. 1-5.

[0051]FIGS. 7A and 7B are flow diagrams of process 700 to determine an impact vector, according to one or more embodiments.

[0052]Process 700 begins by receiving a series of images of a scene from an imager, e.g., a video camera (702). Process 700 selects N images (N=3) from the series of images that each contain the striking object (e.g., golf club) before the striking object comes into contact with a ball (e.g., a golf ball) (704). It is understood that process 700 may select more than three images that each contain the striking object before the striking object contacts the ball, for example four, five, six images. Process 700 feeds the three images the trained machine learning model to determine the normal vector and center position of the golf club in the N images (708). Process 700 continues by applying a circle fitting algorithm to the N positions to determine if the three positions fit into a circle (710). This assumes that when the golfer swings the golf club, the swing is substantially circular motion. The circle fitting algorithm determines if the positions P₋₁, P₋₂ and P₋₃ are in the same images of the same swing. If not, then process 700 will select another three images. If yes, process 700 has determined that the center position of the golf club from the three selected images are of the same swing in the series of images.

[0053]Process 700 continues by selecting the image of the club that is the closest to the ball (714), determining a position when the ball is stationary from the series of images (716), determining the impact location where the ball is stationary from the three selected pre-impact images, and selecting a pre-impact image that is closest to the ball (718). Each of these steps was previously described in reference to FIGS. 1-5.

[0054]FIG. 8 illustrates determining the impact location where a detection unit 802 is placed behind a user 804, according to one or more embodiments. In the example shown, user 804 is preparing to swing golf club 803 and aim the ball onto an area 806 close to the hole. Detection unit 802 comprising one or more imagers captures a series of images of the swing and determines the impact location as described in reference to FIGS. 1-7. An example detection unit 802 can be the mobile launch monitor or MLM, or any other similar devices that is capable of capturing images and processing images.

[0055]FIG. 9 illustrates determining an impact location in an indoor setting where a detection unit 904 is placed above user 803 according to one or more embodiments. In some embodiments, the detection unit 904 is a ceiling-mounted detection unit. As above, the detection unit 904 comprises one or more imagers. Detection unit 904 can be the MLM., or any other similar devices that is capable of capturing images and processing images and simulation. As shown, user 804 is preparing to swing golf club 803 and aim the ball onto screen 906, where projector 902 projects a virtual image of a golf course. It is understood that FIGS. 8 and 9 are mere examples of placement of the detection unit and therefore other suitable placements may also be envisioned.

[0056]FIG. 10 is a conceptual block diagram of detection unit 802 for determining the impact location according to one or more embodiments. Detection unit 802 includes at least one processor 1002, computer memory 1012 including machine learning model/program instructions 1010 stored thereon, camera 1004 and a radar 1006 as described in reference to FIGS. 1-9. Detection unit 802 described above is one example embodiment of a suitable processing architecture. Other suitable processing architectures can also be used to implement the embodiments described herein.

[0057]While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a sub combination or variation of a sub combination.

[0058]Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.