LookaheadOptimizer

注意：该API仅支持【静态图】模式

  • class paddle.fluid.optimizer.LookaheadOptimizer(inner_optimizer, alpha=0.5, k=5)[源代码]

本类实现了Lookahead优化算法：https://arxiv.org/abs/1907.08610。Lookahead优化算法在内存中保存两部分参数：快参数和慢参数。每个训练步次，inner_optimizer都更新快参数；每隔k个训练步次，Lookahead更新慢参数，如下：

LookaheadOptimizer - 图1

  • 参数:
    • inner_optimizer (Optimizer) - 基础优化器，如SGD
    • alpha (float) - Lookahead 的学习率
    • k (int) - 慢参数更新的频率：k次一更新

代码示例

  1. import paddle
  2. import paddle.fluid as fluid
  3. import numpy as np
  4.  
  5. x = fluid.layers.data(name='x', shape=[2], dtype='float32')
  6. label = fluid.layers.data(name="label", shape=[1], dtype="int64")
  7. y = fluid.layers.fc(input=[x], size=2, act="softmax")
  8. loss = fluid.layers.cross_entropy(input=y, label=label)
  9. loss = fluid.layers.mean(x=loss)
  10. sgd = fluid.optimizer.SGD(learning_rate=0.01)
  11. optimizer = fluid.optimizer.LookaheadOptimizer(sgd,
  12.                                 alpha=0.5,
  13.                                 k=5)
  14. optimizer.minimize(loss)
  15. main_program = fluid.default_main_program()
  16. place = fluid.CPUPlace()
  17. exe = fluid.Executor(place)
  18. exe.run(fluid.default_startup_program())
  19.  
  20. feeder = fluid.DataFeeder(feed_list=[x, label], place=place)
  21.  
  22. step = 0
  23. while(step < 10):
  24.     step += 1
  25.     exe.run(fluid.default_main_program(),
  26.     feed=feeder.feed(batch_data))