EngineConn architecture design

EngineConn: Engine connector, a module that provides functions such as unified configuration management, context service, physical library, data source management, microservice management, and historical task query for other microservice modules.

EngineConn architecture diagram

EngineConn

Introduction to the second-level module:

linkis-computation-engineconn interactive engine connector

The ability to provide interactive computing tasks.

Core classCore function
EngineConnTaskDefines the interactive computing tasks submitted to EngineConn
ComputationExecutorDefined interactive Executor, with interactive capabilities such as status query and task kill.
TaskExecutionServiceProvides management functions for interactive computing tasks

linkis-engineconn-common engine connector common module

Define the most basic entity classes and interfaces in the engine connector. EngineConn is used to create a connection session for the underlying computing storage engine, which contains the session information between the engine and the specific cluster, and is the client that communicates with the specific engine.

Core ServiceCore function
EngineCreationContextContains the context information of EngineConn during startup
EngineConnContains the specific information of EngineConn, such as type, specific connection information with layer computing storage engine, etc.
EngineExecutionProvide Executor creation logic
EngineConnHookDefine the operations before and after each phase of engine startup

The core logic of linkis-engineconn-core engine connector

Defines the interfaces involved in the core logic of EngineConn.

Core classCore function
EngineConnManagerProvide related interfaces for creating and obtaining EngineConn
ExecutorManagerProvide related interfaces for creating and obtaining Executor
ShutdownHookDefine the operation of the engine shutdown phase

linkis-engineconn-launch engine connector startup module

Defines the logic of how to start EngineConn.

Core classcore function
EngineConnServerEngineConn microservice startup class

The core logic of the linkis-executor-core executor

Defines the core classes related to the actuator. The executor is a real computing scene executor, responsible for submitting user code to EngineConn.

Core classCore function
ExecutorIt is the actual computational logic execution unit and provides a top-level abstraction of the various capabilities of the engine.
EngineConnAsyncEventDefines EngineConn-related asynchronous events
EngineConnSyncEventDefines EngineConn-related synchronization events
EngineConnAsyncListenerDefines EngineConn related asynchronous event listener
EngineConnSyncListenerDefines EngineConn related synchronization event listener
EngineConnAsyncListenerBusDefines the listener bus for EngineConn asynchronous events
EngineConnSyncListenerBusDefines the listener bus for EngineConn synchronization events
ExecutorListenerBusContextDefines the context of the EngineConn event listener
LabelServiceProvide label reporting function
ManagerServiceProvides the function of information transfer with LinkisManager

linkis-callback-service callback logic

Core ClassCore Function
EngineConnCallbackDefine EngineConn’s callback logic

linkis-accessible-executor can be accessed executor

Executor that can be accessed. You can interact with it through RPC requests to get its status, load, concurrency and other basic indicators Metrics data.

Core ClassCore Function
LogCacheProvide log cache function
AccessibleExecutorThe Executor that can be accessed can interact with it through RPC requests.
NodeHealthyInfoManagerManage Executor’s Health Information
NodeHeartbeatMsgManagerManage the heartbeat information of Executor
NodeOverLoadInfoManagerManage Executor load information
ListenerProvides events related to Executor and the corresponding listener definition
EngineConnTimedLockDefine Executor level lock
AccessibleServiceProvides the start-stop and status acquisition functions of Executor
ExecutorHeartbeatServiceProvides heartbeat related functions of Executor
LockServiceProvide lock management function
LogServiceProvide log management functions