Overview

Overview

Background

BML (Material Library Service) is a material management system of linkis, which is mainly used to store various file data of users, including user scripts, resource files, third-party Jar packages, etc., and can also store class libraries that need to be used when the engine is running.

It has the following functions:

1) Support various types of files. Supports text and binary files. If you are a user in the field of big data, you can store their script files and material compression packages in the system.

2), the service is stateless, multi-instance deployment, to achieve high service availability. When the system is deployed, it can be deployed with multiple instances. Each instance provides services independently to the outside world without interfering with each other. All information is stored in the database for sharing.

3) Various ways of use. Provides two ways of Rest interface and SDK, users can choose according to their needs.

4) The file is appended to avoid too many small HDFS files. Many small HDFS files will lead to a decrease in the overall performance of HDFS. We have adopted a file append method to combine multiple versions of resource files into one large file, effectively reducing the number of files in HDFS.

5) Accurate authority control, safe storage of user resource file content. Resource files often have important content, and users only want to read it by themselves

6) Provide life cycle management of file upload, update, download and other operational tasks.

Architecture diagram

Schema description

The Service layer includes resource management, uploading resources, downloading resources, sharing resources, and project resource management.

Resource management is responsible for basic operations such as adding, deleting, modifying, and checking resources, controlling access rights, and whether files are out of date.

File version control Each BML resource file has version information. Each update operation of the same resource will generate a new version. Of course, it also supports historical version query and download operations. BML uses the version information table to record the deviation position and size of each version of the resource file HDFS storage, and can store multiple versions of data on one HDFS file.
Resource file storage HDFS files are mainly used as actual data storage. HDFS files can effectively ensure that the material library files are not lost. The files are appended to avoid too many small HDFS files.

Core Process

upload files:

Determine the operation type of the file uploaded by the user, whether it is the first upload or update upload. If it is the first upload, a new resource information record needs to be added. The system has generated a globally uniquely identified resource_id and a resource_location for this resource. The first version A1 of resource A needs to be stored in the resource_location location in the HDFS file system. After storing, you can get the first version marked as V00001. If it is an update upload, you need to find the latest version last time.
Upload the file stream to the specified HDFS file. If it is an update, it will be added to the end of the last content by file appending.
Add a new version record, each upload will generate a new version record. In addition to recording the metadata information of this version, the most important thing is to record the storage location of the version of the file, including the file path, start location, and end location.

download file:

When users download resources, they need to specify two parameters: one is resource_id and the other is version. If version is not specified, the latest version will be downloaded by default.
After the user passes in the two parameters resource_id and version to the system, the system queries the resource_version table, finds the corresponding resource_location, start_byte and end_byte to download, and uses the skipByte method of stream processing to set the front (start_byte- 1) skip bytes, then read to end_byte The number of bytes. After the reading is successful, the stream information is returned to the user.
Insert a successful download record in resource_download_history

Database Design

Resource information table (resource)

Field name	Function	Remarks
resource_id	A string that uniquely identifies a resource globally	UUID can be used for identification
resource_location	The location where resources are stored	For example, hdfs:///tmp/bdp/\${USERNAME}/
owner	The owner of the resource	e.g. zhangsan
create_time	Record creation time
is_share	Whether to share	0 means not to share, 1 means to share
update_time	Last update time of the resource
is_expire	Whether the record resource expires
expire_time	Record resource expiration time

Resource version information table (resource_version)

Field name	Function	Remarks
resource_id	Uniquely identifies the resource	Joint primary key
version	The version of the resource file
start_byte	Start byte count of resource file
end_byte	End bytes of resource file
size	Resource file size
resource_location	Resource file placement location
start_time	Record upload start time
end_time	End time of record upload
updater	Record update user

Resource download history table (resource_download_history)

Field	Function	Remarks
resource_id	Record the resource_id of the downloaded resource
version	Record the version of the downloaded resource
downloader	Record downloaded users
start_time	Record download time
end_time	Record end time
status	Whether the record is successful	0 means success, 1 means failure
err_msg	Log failure reason	null means success, otherwise log failure reason