Mixed reality based 3D sketching device and method

CROSS-REFERENCE TO RELATED APPLICATION

[0001]This present application claims priority under 35 U.S.C. § 119(a) to Korean Patent Application No. 10-2019-0114815 filed on Sep. 18, 2019, the entire contents of which are incorporated herein by reference.

BACKGROUND

(a) Technical Field

[0002]Embodiments of the present disclosure relate to a mixed reality based 3D sketching device and method.

(b) Background Art

[0003]Recently, various attempts using virtual reality (VR) technology are being made.

[0004]Alibaba provides an opportunity to experience a virtual shopping mall, thus showing a possibility that a virtual space will replace a real space. Google has released a tool called Tilt Brush that may perform a 3D sketch and modeling on the VR, thus greatly expanding the range of creative means. Furthermore, attempts are made to increase portability and accessibility by grafting the idea of the VR technology with smartphone-based augmented reality (AR) technology.

[0005]An app such as Just a Line is an example of reconfiguring the 3D sketch in the form of AR. However, existing 3D sketching systems have limitations. A VR-based sketching system is limited in accessibility and portability, and is impossible in multitasking.

[0006]Just a Line solving the above problems is configured to sketch its own trajectory by an Inertial Measurement Unit (IMU) installed in the smartphone. This is problematic in that a pen and a viewer work integrally without being separated from each other, so that it is not useful for anything other than games.

[0007]Furthermore, a Ramer-Douglas-Peucker algorithm is mainly used so as to minimize the number of points in the 3D sketch on the conventional augmented reality, but is not suitable to minimize the number of points in real time.

SUMMARY OF THE DISCLOSURE

[0008]The present disclosure has been made to solve the above-mentioned problems and difficulties and relates to a mixed reality based 3D sketching device and method that can increase work efficiency and can effectively reduce the number of data.

[0009]According to one aspect of the disclosure, the present disclosure provides a mixed reality based 3D sketching device, including a processor; and a memory connected to the processor, wherein the memory stores program commands that are executable by the processor to periodically tracks a marker pen photographed through a camera, to determine whether to remove a third point using a distance between a first point corresponding to a reference point, among points that are sequentially tracked, and a second point at a current time, a preset constant, and an angle between the first point, the second point, and the previously identified third point, to search an object model corresponding to a 3D sketch that has been corrected, after correction is completed depending on the removal of the third point, and to display the searched object model on a screen.

[0010]The marker pen may be a cube-type marker pen configured so that a marker may be attached to an entire outer surface of a cylinder structure.

[0011]The removal of the third point may be determined by the following Equation.
if Θ_(x,p)<α/dist(o,p) then Remove(x)  [Equation]

[0012]where o is a reference point, p is a second point, and x is a third point.

[0013]The program commands may orthogonally project the 3D sketch that has been corrected into a 2D image, thus extracting a characteristic value, and may compare the extracted characteristic value with a previously stored 3D object model, thus searching a 3D object model corresponding to the 3D sketch.

[0014]According to another aspect of the disclosure, the present disclosure provides a mixed reality based 3D sketching method of performing a mixed reality based 3D sketch in a device including a processor and a memory, the method including periodically tracking a marker pen photographed through a camera; determining whether to remove a third point using a distance between a first point corresponding to a reference point, among points that are sequentially tracked, and a second point at a current time, a preset constant, and an angle between the first point, the second point, and the previously identified third point, searching an object model corresponding to a 3D sketch that has been corrected, after correction is completed depending on the removal of the third point, and displaying the searched object model on a screen.

[0015]According to a further aspect of the disclosure, the present disclosure provides a program for performing the above-described method.

[0016]According to the present disclosure, it is possible to track a marker even if a portion of a pen is visible through a cube-type marker pen and to minimize the number of points in real time by correction through an included angle.

BRIEF DESCRIPTION OF THE DRAWINGS

[0017]FIG. 1 is a diagram illustrating a mixed reality based 3D sketching system in accordance with an embodiment of the present disclosure.

[0018]FIG. 2 is a diagram illustrating a mixed reality based service in accordance with the present embodiment.

[0019]FIG. 3 is a diagram illustrating a configuration of a cube-type marker pen in accordance with an embodiment of the present disclosure.

[0020]FIG. 4 is a diagram illustrating a process of determining whether to add a new point x through an included angle in accordance with the present embodiment.

[0021]FIG. 5 is a diagram illustrating a case where the new point x is removed.

[0022]FIG. 6 is a diagram illustrating a case where the new point x is not removed.

[0023]FIGS. 7A and 7B are a diagram illustrating a correcting process in accordance with the present embodiment.

[0024]FIGS. 8A and 8B are a diagram illustrating a process of searching and inserting an object model corresponding to a 3D sketch in accordance with the present embodiment.

[0025]FIG. 9 is a diagram illustrating a configuration of a user terminal in accordance with an embodiment of the present disclosure.

DETAILED DESCRIPTION

[0026]Since the present disclosure may be embodied in many different forms, specific embodiments are illustrated in the drawings and described in detail.

[0027]However, the disclosure should not be construed as limited to the embodiments set forth herein. Rather, it is to be understood that all changes, equivalents, and substitutes included in the spirit and scope of the disclosure fall within the purview of the disclosure.

[0028]FIG. 1 is a diagram illustrating a mixed reality based 3D sketching system in accordance with an embodiment of the present disclosure.

[0029]As illustrated in FIG. 1, the 3D sketching system in accordance with this embodiment may include a service server 100 and a user terminal 102.

[0030]The service server 100 and the user terminal 102 may be connected via a network. Here, the network may include, for example, an internet network, a mobile communication network, etc.

[0031]FIG. 2 is a diagram illustrating a mixed reality based service in accordance with the present embodiment.

[0032]As illustrated in FIGS. 1 and 2, the user terminal 102 in accordance with this embodiment may include a host terminal that creates a shared mixed reality space, and one or more guest terminals that creates and edits an object applied to the shared mixed reality space created by the host terminal through a 3D sketch.

[0033]Of course, the host terminal may also create and edit the object.

[0034]The user terminal 102 in accordance with this embodiment may include a mobile communication terminal (smartphone), a tablet, a desktop, and a laptop in which an application (app) for creating and editing the object is installed. Hereinafter, an example where the user terminal is the smartphone will be described.

[0035]The app installed in the user terminal 102 is operated in conjunction with a pen including a marker to allow a trajectory sketched on a screen by a user to be displayed and edited.

[0036]Furthermore, the app in accordance with this embodiment creates a mixed reality space through a mobile augmented reality platform and simultaneously combines the mixed reality space and a real space through a camera, thus implementing an augmented reality.

[0037]The app in accordance with this embodiment may output an interface for the 3D sketch. The interface may include the shape, size, and color of the pen, sketch-related functions such as undo, redo, and erase, and functions that may be used after the sketch is completed, such as edit, save, load, model search, share, and cloud anchor (cloud anchor connection).

[0038]A marker pen in accordance with this embodiment is a cube-type pen to allow the marker to be easily recognized.

[0039]The service server 100 is connected to a plurality of user terminals 102 to provide a mixed reality service.

[0040]The service server 100 previously stores a plurality of 3D models applied to the mixed reality space. In the case of creating the 3D sketch in the user terminal 102, a corresponding object is searched and then is displayed on the mixed reality space.

[0041]Furthermore, the service server 100 may store coordinate information about the mixed reality space in which the host terminal is created, and share-related information (e.g., room code). Other guest terminals may create and edit the object displayed on the mixed reality space through the share-related information.

[0042]FIG. 3 is a diagram illustrating the configuration of the cube-type marker pen in accordance with an embodiment of the present disclosure.

[0043]As illustrated in FIG. 3, the cube-type marker pen in accordance with this embodiment may be configured such that the marker is attached to an entire cylinder structure, and may include a button and a bluetooth module.

[0044]The button and the bluetooth module are used to start/stop a sketch work.

[0045]Conventionally, a hexahedral marker is provided on an end of the pen. However, if the marker is provided on the entire pen as disclosed in this embodiment, it is possible to track the marker even when a portion of the pen is visible.

[0046]The marker in accordance with this embodiment may be recognized through the camera provided in the smartphone. However, without being limited thereto, the camera installed in an outside instead of the smartphone may be placed in a proper location to recognize the marker. The smartphone may serve as only a viewer.

[0047]According to an embodiment of the present disclosure, in the case of performing the 3D sketch through marker tracking, a new technique is proposed to minimize the number of points.

[0048]FIG. 4 is a diagram illustrating a process of determining whether to add a new point x through an included angle in accordance with the present embodiment.

[0049]Referring to FIG. 4, in a process of adding the new point x(t=1) where a previous frame is tracked, an included angle θ between x and a current pen position p(t=2) with reference to a previously written point o(t=0) is calculated, and it is determined whether the included angle θ is smaller than a value obtained by dividing a preset constant α by a distance d between o and p.

[0050]This may be expressed by the following equation.
if Θ_(x,p)<α/dist(o,p) then Remove(x)  [Equation 1]

[0051]Here, a removal allowable range can be adjusted by adjusting the preset constant α.

[0052]FIG. 5 is a diagram illustrating a case where the new point x is removed.

[0053]Referring to FIG. 5, if the included angle θ is smaller than a value obtained by dividing the preset constant α by the distance d between o and p, this is considered as noise and x is removed.

[0054]That is, since the point x of the previous frame has an angle difference smaller than a point p where the pen is currently located, the user terminal 102 considers x as noise and performs correction to remove x.

[0055]In contrast, referring to FIG. 6, if the point x of the previous frame has an angle difference larger than the point p where the pen is currently located, x is not considered as noise, so that x is not removed.

[0056]In the case of performing the correction using the included angle according to this embodiment, as illustrated in FIGS. 7A and 7B, both the correction and the minimization of the point may be performed even in a line that is finely bent due to a user's hand trembling.

[0057]FIGS. 8A and 8B are a diagram illustrating a process of searching and inserting an object model corresponding to a 3D sketch in accordance with the present embodiment.

[0058]After the 3D sketch is completed using the cube-type marker pen, the user terminal 102 according to this embodiment orthogonally projects the 3D sketch into the 2D image depending on where the camera is looking, thus extracting a characteristic value. By using it as an index, a 3D object model stored in a database is searched.

[0059]Here, the 3D object model may be searched in conjunction with the service server 100.

[0060]A user may attempt a search until a desired model is found while continuously changing a direction in which he or she is looking. For each search, a system returns n models of a high rank in the order of matching of images used in the search and characteristic values. If the search is executed, the most similar models are shown on the screen as illustrated in FIG. 8A. If one of the models is selected, the model is inserted into the mixed reality space as illustrated in FIG. 8B.

[0061]According to this embodiment, the object completed through the app may be filed to be stored in or loaded from the service server 100 and be shared with others.

[0062]Since the database of the service server 100 is managed on an account basis, a user may perform the work more conveniently through a user's own account.

[0063]In order to share the mixed reality space, Cloud Anchors of Google may be used, for example.

[0064]The mixed reality space is created based on an anchor. The system according to this embodiment may be composed of a host that provides information about the anchor and guests that receive the information.

[0065]If the user of the host terminal taps on a specific anchor on the app, the information of the anchor tapped on the database and his or her own Room Code are stored. As illustrated in FIG. 8b, the guests may access the anchor shared through the Room Code.

[0066]FIG. 9 is a diagram illustrating the configuration of the user terminal in accordance with an embodiment of the present disclosure.

[0067]As illustrated in FIG. 9, the user terminal in accordance with this embodiment may include a processor 900 and a memory 902.

[0068]The processor 900 may, for example, include a central processing unit (CPU) that may execute a computer program, other virtual machines, etc.

[0069]The memory 902 may, for example, include a non-volatile storage device such as a fixed hard drive or a removable storage device. The removable storage device may, for example, include a compact flash unit, a USB memory stick, etc. The memory 902 may include a volatile memory such as various kinds of random access memories.

[0070]According to an embodiment of the present disclosure, the processor 900 periodically tracks the cube-type marker pen photographed through the camera using program commands stored in the memory 902, and identifies each tracked point.

[0071]The program commands according to this embodiment use an included angle Θ_{(x, p)} of three points (o, x, p in t=0 to t=2) that are sequentially tracked, a preset constant α, and a distance d dist(o, p) between two points to determine whether to remove a preceding point (x in t=1).

[0072]The removal of x is continuously performed according to the tracking. When the 3D sketch is completed by the correction, this is orthogonally projected into the 2D image and then a characteristic value is extracted.

[0073]The extracted characteristic value is compared with the 3D object model that is previously stored, and the 3D object model corresponding to the 3D sketch is searched and then output on the screen.

[0074]Furthermore, the program commands according to this embodiment create the mixed reality space, and the created mixed reality space is stored in the database connected to the service server 100.

[0075]Guest terminals may access the mixed reality space created by another user through the anchor, and a new object may be created and edited through the 3D sketch in the accessed mixed reality space.

[0076]Although the present disclosure was described with reference to specific embodiments shown in the drawings, it is apparent to those skilled in the art that the present disclosure may be changed and modified in various ways without departing from the scope of the present invention.