Basic Concept

NameDescriptionOther
GroupData Streams Group, it contains multiple data streams, and one Group represents one data ingestion.Group has attributes such as ID and Name.
StreamData Stream, a stream has a specific flow direction.Stream has attributes such as ID, Name, and data fields.
InLongMsgInLong data format, if you consume message directly from the message queue, you need to perform InLongMsg parsing first.
AgentRepresents various collection capabilities.It contains File Agent, SQL Agent, Binlog Agent, etc.
DataProxyForward received data to different message queues.Supports data transmission blocking, placing retransmission.
SortData stream sortingSort-flink based on Flink, sort-standalone for local sorting.
TubeMQInLong’s self-developed message queuing serviceIt can also be called Tube, with low-cost, high-performance features.
PulsarApache Pulsar, a high-performance, high-consistency message queue service
HiveApache Hive, a data warehouse built on the Hadoop architecture
ClickHouseClickHouse, a high performance columnar OLAP database