D5AI LLC

A computer system uses a pool of predefined functions and pre-trained networks to accelerate the process of building a large neural network or building a combination of (i) an ensemble of other machine learning systems with (ii) a deep neural network. Copies of a predefined function node or network may be placed in multiple locations in a network being built. In building a neural network using a pool of predefined networks, the computer system only needs to decide the relative location of each copy of a predefined network or function. The location may be determined by (i) the connections to a predefined network from source nodes and (ii) the connections from a predefined network to nodes in an upper network. The computer system may perform an iterative process of selecting trial locations for connecting arcs and evaluating the connections to choose the best ones.

Systems and methods analyze and correct the vulnerability of individual nodes in a neural network to changes in the input data. The analysis comprises first changing the activation function of one or more nodes to make them more vulnerable. The vulnerability is then measured based on a norm on the vector of partial derivatives of the network objective evaluated on each training data item. The system is made less vulnerable by splitting the data based on the sign of the partial derivative of the network objective with respect to a vulnerable and training new ensemble members on selected subsets from the data split.

Computer-implemented systems and methods soft-tie learned parameters of a neural network(s). The soft-tying comprises: applying a common label to the first and second learned parameters; and as part of the training, and in response to the first and second learned parameters having the common label, applying a regularization penalty to a loss function for the first learned parameter upon a determination that the first learned parameter is different than the second learned parameter. The learned parameters can be connection weights, node biases, and/or parametric model statistics. The application of the regularization penalty can be influenced by a soft-tying hyperparameter.

Computer systems and methods optimize a secondary objective function in the training of a multi-layer feed-forward neural network in which the secondary objective is a function of the partial derivatives of the primary objective function. Optimizing this secondary objective function comprises computing derivatives of functions of the partial derivatives computed during the back-propagation computation in a third stage of computation before the parameter update. This third stage of computation proceeds in the reverse direction from the direction of the back propagation computation. That is, the third stage of computation proceeds forwards through the network, computing derivatives of the secondary objective function based on the chain rule of calculus. The secondary objective may be used to make the neural network more robust against deviations in the input values from their normal values.