Ab Initio Technology LLC

An approach to allocation of referenced objects to memory resources addresses a situation in which there are a far greater number of memory resources, for example, 2 16 elements in the set of memory resources, and yet the objects referenced in a program specification exceeds this number. The approach is applicable to compilation of a program specification for execution on a physical or virtual processor.

A method for using a development environment to automatically generate code from a multi-tiered metadata model includes: receiving a specification to process a dataset, and, in response, accessing dataset characteristics and identifying controls received from a development environment to be applied to a field of the dataset in accordance with a metadata model by: accessing a first instance of a data structure that corresponds to the dataset; based on a reference in the first instance, accessing a second instance of a data structure associated with the field; based on a reference in the second instance, accessing a third instance of a data structure associated with metadata describing the field, and based on a reference in the third instance, accessing a fourth instance of a data structure storing a control defined based on the metadata. Based on the dataset characteristics, code is generated to apply the identified control to the field.

A data processing system with a dataset multiplexer that enables applications to be written to specify access to datasets as operations on logical datasets. During execution of an application by the data processing system, operations that access a dataset are implemented by accessing an entry in a dataset catalog for the logical dataset. That entry includes information to access the physical data source storing the logical dataset, including conversion of data from the format of the physical data source to the format of the logical dataset. An entry in the catalog may be created based on registration of a data source with the dataset multiplexer and may be updated automatically based on changes in storage of the dataset. This maintenance of the catalog may be partially or totally automated such that the system automatically adjusts to any changes in storage of the dataset without need for modification of any application.

A method implemented by a data processing system for enabling a user to browse a data catalog and select fields from multiple data sources to be integrated into a data profile so that, when a request is received for the data profile, data from those fields can be made available efficiently and immediately.

A method for fault-tolerant processing of a number of data elements using a distributed computing cluster. The distributed computing cluster includes a number of data processors associated with a corresponding number of data stores. The method includes storing the data elements in the distributed computing cluster, wherein the data elements are distributed across the data stores according to a number of partitions of data elements, processing data elements of a first set of partitions stored at a first data store using a first data processor to generate first result data for the data elements of the first set of partitions, sending the first result data from the distributed computing cluster to a consumer of the first result data outside the distributed computing cluster, and storing the first result data in a first buffer located in the distributed computing cluster and associated with the first data processor until the consumer has persistently stored the first result data outside the distributed computing cluster.

A data processing system with a dataset multiplexer that enables applications to be written to specify access to datasets as operations on logical datasets. During execution of an application by the data processing system, the physical dataset used for performing data access operations may be selected based on current context. Current context may be specified based on values of system parameters and/or user specified values. The physical dataset accessed may be identified by selecting a record from multiple records in a dataset catalog associated with the logical dataset. Each record includes information to access a physical dataset associated with the selected record and context information to indicate the context in which the specific physical dataset is to be selected.

A computer-implemented method for defining a test for a computer program includes receiving operational data generated during execution of a computer program in a first computing environment, the operational data indicative of (i) a data source accessed by the computer program during execution of the computer program and (ii) a destination to where baseline data records are output by the computer program during execution of the computer program. Based on the received operational data, a data storage object is generated that includes (i) input data records from the data source and the baseline data records from the destination, and (ii) test definition data for the first computing environment. Responsive to migration of the computer program to a second computing environment, the input and baseline data records from the data storage object are stored in the second computing environment. A test configuration is defined for the migrated computer program in the second computing environment according to the test definition data in the data storage object and the mapping between the first computing environment and the second computing environment, the test configuration for the migrated computer program identifying a location of the input data records and a location of the baseline data records in the second computing environment. Execution of the migrated computer program in the second computing environment is tested using the input data records and baseline data records in the second computing environment and according to the defined test configuration for the migrated computer program.

Some embodiments relate to generating a list of data fields referenceable at a point in a graph (there are different lists for each point). This list may be used as part of programming a dataflow graph to select data (e.g., at an input node of a component to select data processed in that component). One aspect relates to display of the list of data fields, because some of the data field names may be overloaded. Accordingly, the data fields may be presented hierarchically if necessary, showing the source for each overloaded data field name. Otherwise, the user may select whether the list of referenceable fields is grouped by source.

A method implemented by a data processing system including: accessing the container image that includes the first application and a second application; determining, by the data processing system, the number of parallel executions of the given module of the first application; for the given module, generating a plurality of instances of the container image in accordance with the number of parallel executions determined, for each instance, configuring that instance to execute the given module of the first application; causing each of the plurality of configured instances to execute on one or more of the host systems; and for at least one of the plurality of configured instances, causing, by the second application of that configured instance, communication between the data processing system and the one or more of the host systems executing that configured instance.

Some embodiments relate to a method for use in connection with governance of a plurality of data assets managed by a data processing system, the method comprising: using at least one computer hardware processor to perform: accessing a data governance policy comprising a first data standard (e.g., by obtaining information about the first standard stored in a database system); generating a first data asset collection at least in part by automatically selecting, from among the plurality of data assets managed by the data processing system and using at least one data asset criterion, one or more data assets that meet the at least one data asset criterion; associating the first data asset collection with the first data standard; and verifying whether at least one of the one or more data assets in the first data asset collection complies with the first data standard.