Linkis内置时间变量介绍

1.总述

需求背景

用户希望在写代码时,对时间的格式要求千变万化,已有的Linkis自定义变量目前还不足以支撑这些需求。另外,已有的时间运算-1有些表示减一个月,有些则是减一天,用户很容易混淆

目标

  • 其他日期内置变量都是相对run_date计算出来
  • 支持Pattern格式时间且用户可以自行随意指定
  • 支持±y/±M/±d/±H等等

Pattern格式对照表:

LetterDate or Time ComponentPresentationExamples
GEra designatorTextAD
yYearYear1996; 96
YWeek yearYear2009; 09
MMonth in yearMonthJuly; Jul; 07
wWeek in yearNumber27
WWeek in monthNumber2
DDay in yearNumber189
dDay in monthNumber10
FDay of week in monthNumber2
EDay name in weekTextTuesday; Tue
uDay number of week (1 = Monday, …, 7 = Sunday)Number1
aAm/pm markerTextPM
HHour in day (0-23)Number0
kHour in day (1-24)Number24
KHour in am/pm (0-11)Number0
hHour in am/pm (1-12)Number12
mMinute in hourNumber30
sSecond in minuteNumber55
SMillisecondNumber978
zTime zoneGeneral time zonePacific Standard Time; PST; GMT-08:00
ZTime zoneRFC 822 time zone-0800
XTime zoneISO 8601 time zone-08; -0800; -08:00

2. 总体设计

总体设计和技术架构参照Linkis自定义变量

3. 功能介绍

  • Linkis支持的变量类型分为自定义变量(不做赘述)和系统内置变量,自定变量日期支持+-。
  • 其中+-是先对linkis内置参数run_date进行运算,然后将%之前的pattern字段进行替换,非Pattern字符不支持运算替换。

3.1 内置变量举例

可以根据自己的喜好/业务实际情况定义需要动态渲染的参数

variableresult
&{yyyy-01-01}2021-01-01
&{yyyy-01-01%-2y}2019-01-01
&{yyyy-MM-01%-2M}2021-02-01
&{yyyy-MM-dd%-2d}2021-03-31
&{yyyy MM ——- HH%-1H}2021 04 ——- 14
&{yyyyMMdd%-1d}20210401
&{yyyyMM01%-1M}20210301
&{HH%-1H}14

3.2 自定义变量使用实例

  • 例子1: sql
  1. SELECT * FROM hive.tmp.fund_nav_histories
  2. WHERE dt <= DATE_FORMAT(DATE_ADD('day', -1, DATE(Date_parse('&{yyyyMMdd%-1d}', '%Y%m%d'))), '%Y%m%d')

渲染后

  1. SELECT * FROM hive.tmp.fund_nav_histories
  2. WHERE dt <= DATE_FORMAT(DATE_ADD('day', -1, DATE(Date_parse('20220705', '%Y%m%d'))), '%Y%m%d')
  • 例子2: shell
  1. aws s3 ls s3://***/ads/tmp/dws_member_active_detail_d_20210601_20211231/pt=&{yyyyMMdd%-1d}/

渲染后

  1. aws s3 ls s3://***/ads/tmp/dws_member_active_detail_d_20210601_20211231/pt=20220705/
  • 例子3: datax json
  1. {
  2. "job": {
  3. "setting": {
  4. "speed": {
  5. "channel": 1
  6. }
  7. },
  8. "content": [
  9. {
  10. "reader": {
  11. "name": "s3reader",
  12. "parameter": {
  13. "bucket": "**************",
  14. "path": [
  15. "ads/tmp/ccass_tm_announcements/&{yyyyMMdd%-1d}/"
  16. ],
  17. "stored": "parquet",
  18. "compression": "NONE",
  19. "column": [
  20. {
  21. "index": 0,
  22. "type": "int"
  23. },
  24. {
  25. "index": 1,
  26. "type": "string",
  27. "constant": "&{yyyyMMdd%-1d}"
  28. }
  29. ]
  30. }
  31. },
  32. "writer": {
  33. "name": "streamwriter",
  34. "parameter": {
  35. "print": true
  36. }
  37. }
  38. }
  39. ]
  40. }
  41. }

渲染后

  1. {
  2. "job": {
  3. "setting": {
  4. "speed": {
  5. "channel": 1
  6. }
  7. },
  8. "content": [
  9. {
  10. "reader": {
  11. "name": "s3reader",
  12. "parameter": {
  13. "bucket": "**************",
  14. "path": [
  15. "ads/tmp/ccass_tm_announcements/20220705/"
  16. ],
  17. "stored": "parquet",
  18. "compression": "NONE",
  19. "column": [
  20. {
  21. "index": 0,
  22. "type": "int"
  23. },
  24. {
  25. "index": 1,
  26. "type": "string",
  27. "constant": "20220705"
  28. }
  29. ]
  30. }
  31. },
  32. "writer": {
  33. "name": "streamwriter",
  34. "parameter": {
  35. "print": true
  36. }
  37. }
  38. }
  39. ]
  40. }
  41. }
  • 例子4:python
  1. print(&{yyyyMMdd%-1d})

渲染后

  1. 20220705