Databricks, Inc.

A data processing service receives indication that a recipient will request access to data assets of a provider and provides a request for credentials from a recipient governance module. The recipient governance module stores a recipient metastore including an object for a provider metastore. In response to determining that the assets are associated with the provider metastore, the service provides a request for credentials to a provider governance module. The provider governance module stores the provider metastore describing data assets of the provider and permissions for accessing data assets. The provider metastore includes a recipient object attached to the data assets with an identifier for the recipient metastore. In response to verifying that the recipient was provided access to the data assets, the service provides a token to the recipient governance module. The service then provides the token to a computing resource to provide access to the data assets.

A semantic search between a vector embedding of a sample query and vector embeddings of plural historical queries is performed to identify a predetermined number of historical queries that best match the sample query. A model discovery database stores, for each of plural large language models (LLMs) and for each of the plural historical queries, a historical response to the historical query received from the LLM, associated metadata, and a quality rank. For each of the LLMs, a score for each of plural predetermined metrics is determined based on the quality rank of the LLM and the associated metadata in the model discovery database for the identified predetermined number of historical queries. For each of the plural LLMs, an overall score of the LLM is determined based on the determined scores for the plural predetermined metrics. A ranked list of the LLMs is generated based on the overall scores.

A system manages access control for database queries in accordance with access control policies. The techniques may be used for access control of managed clusters in cloud platforms used for data processing, for example, for MapReduce operations. According to an embodiment, the access control policies are fine grained access control policies that allow a user to access a subset of datasets including data of a dataset. The system receives and compiles a database query to generate a query plan for processing the database query. The query plan includes: one or more data access operators, one or more data processing operators, and one or more access control filters. The system executes the query plan using an executor process and a helper process. The helper process executes a data access operator and a corresponding data access filter and the executor process executes one or more data processing operators.

A system replicates database table, for example, from a source database system to a target database system. The system may use change data capture (CDC) for performing replication. The system according to an embodiment, verifies whether the database replication is working correctly. The system ensures that all operations applied to each source database table in the source database table are applied to the corresponding target database table of the target database system. The system repeatedly runs queries to determine whether any changes occurred in the database table since the last time the check was performed. If the system determines changes in the database table, the system partitions the changed rows and processes each partition to determine whether there are any missing inserts and if any updates are corrupted.

A data processing service facilitates the creation and processing of data processing pipelines that process data processing jobs defined with respect to a set of tasks in a sequence and with data dependencies associated with each separate task such that the output from one task is used as input for a subsequent task. In various embodiments, the set of tasks include at least one cleanroom task that is executed in a cleanroom station and at least one non-cleanroom task executed in an execution environment of a user where each task is configured to read one or more input datasets and transform the one or more input datasets into one or more output datasets.

A data processing service generates keys and values for decoder layers. The service receives a request from a client device. The service accesses a ML transformer architecture which includes decoder layers. Each decoder layer is associated with a KV-cache. The service applies the transformer architecture to input tokens obtained from the request to generate output tokens. For each decoder layer, the service generates keys and values for a current decoder layer. The keys and the values for the current decoder layer are generated by applying trained values for a set of parameters to keys and values for a previous decoder layer. The service obtains queries from inputs to the current decoder layer and generates attention outputs for the current decoder layer by combining the queries with the keys and the values for the current decoder layer. The service generates a response to the request from the output tokens.

A data processing service receives a data table which includes a plurality of rows of data. The service assigns the plurality of rows of data to a plurality of computing nodes. The computing nodes belongs to a distributed computing system. At each assigned computing node, the service applies a machine learning (ML) model to the assigned rows of data for indexing each row in the data table. The ML model is trained to determine a set of centroids, and each centroid is associated with a centroid ID. For each row of data in the data table, the service receives a centroid ID of one of the plurality of centroids that the respective row of data is assigned to. The service generates an index of the data table with at least the received centroid ID of each of the plurality of row of data in the data table.

A data processing service schedules execution of operations for shared experts for a MoE-based feed forward network (FFN) of a machine-learning model (e.g., transformer architecture) while all-to-all (A2A) operations for a set of experts are performed for a set of devices (e.g., graphic processor unit (GPU) devices). By scheduling operations of shared experts with the A2A operations, the data processing service may incorporate shared experts without having to schedule additional time and/or resources, leading to shorter processing times and increased computational efficiency.

A system includes an interface, a processor, and a memory. The interface is configured to receive a version of a model from a model registry. The processor is configured to store the version of the model, start a process running the version of the model, and update a proxy with version information associated with the version of the model, wherein the updated proxy indicates to redirect an indication to invoke the version of the model to the process. The memory is coupled to the processor and configured to provide the processor with instructions.

The disclosed configurations provide a method (and/or a computer-readable medium or system) for determining, from a table schema describing keys of a data table, one or more clustering keys that can be used to cluster data files of a data table. The method includes generating features for the data table, generating tokens from the features, generating a prediction for each token by applying to the token a machine-learned transformer model trained to predict a likelihood that the key associated with the token is a clustering key for the data table, determining clustering keys based on the predictions, and clustering data records of the data table into data files based on key-values for the clustering keys.