Event Time

In this section you will learn about writing time-aware Flink programs. Please take a look at Timely Stream Processing to learn about the concepts behind timely stream processing.

For information about how to use time in Flink programs refer to windowing and ProcessFunction.

A prerequisite for using event time processing is setting the right time characteristic. That setting defines how data stream sources behave (for example, whether they will assign timestamps), and what notion of time should be used by window operations like KeyedStream.timeWindow(Time.seconds(30)).

You can set the time characteristic using StreamExecutionEnvironment.setStreamTimeCharacteristic():

  1. final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  2. env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
  3. DataStream<MyEvent> stream = env.addSource(new FlinkKafkaConsumer<MyEvent>(topic, schema, props));
  4. stream
  5. .keyBy( (event) -> event.getUser() )
  6. .timeWindow(Time.hours(1))
  7. .reduce( (a, b) -> a.add(b) )
  8. .addSink(...);
  1. val env = StreamExecutionEnvironment.getExecutionEnvironment
  2. env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
  3. val stream: DataStream[MyEvent] = env.addSource(new FlinkKafkaConsumer[MyEvent](topic, schema, props))
  4. stream
  5. .keyBy( _.getUser )
  6. .timeWindow(Time.hours(1))
  7. .reduce( (a, b) => a.add(b) )
  8. .addSink(...)
  1. env = StreamExecutionEnvironment.get_execution_environment()
  2. env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
  3. # alternatively:
  4. # env.set_stream_time_characteristic(TimeCharacteristic.IngestionTime)
  5. # env.set_stream_time_characteristic(TimeCharacteristic.ProcessingTime)

Note that in order to run this example in event time, the program needs to either use sources that directly define event time for the data and emit watermarks themselves, or the program must inject a Timestamp Assigner & Watermark Generator after the sources. Those functions describe how to access the event timestamps, and what degree of out-of-orderness the event stream exhibits.

Where to go next?

Back to top