GradientClipByNorm

  • class paddle.fluid.clip.GradientClipByNorm(clip_norm)[源代码]

将输入多维Tensor

GradientClipByNorm - 图1 转换为L2范数不超过给定的二范数最大值（ clip_norm ）的多维Tensor。（多维Tensor不是从该类传入， 而是通过 fluid.program_guard 的 main_program 参数传入）。

该类限制了输入多维Tensor

GradientClipByNorm - 图2 的L2范数不会超过 clip_norm 。

GradientClipByNorm - 图3

其中

GradientClipByNorm - 图4 代表 GradientClipByNorm - 图5 的L2范数

GradientClipByNorm - 图6

  • 参数:
    • clip_norm (float) - 二范数最大值

代码示例

  1. import paddle.fluid as fluid
  2. import paddle.fluid.core as core
  3. import paddle
  4. place = core.CPUPlace()
  5. prog = fluid.framework.Program()
  6. startup_program = fluid.framework.Program()
  7. with fluid.program_guard(
  8.             main_program=prog, startup_program=startup_program):
  9.     image = fluid.layers.data(name='x', shape=[784], dtype='float32')
  10.     label = fluid.layers.data(name='y', shape=[1], dtype='int64')
  11.     hidden1 = fluid.layers.fc(input=image, size=128, act='relu')
  12.     hidden2 = fluid.layers.fc(input=hidden1, size=64, act='relu')
  13.     predict = fluid.layers.fc(input=hidden2, size=10, act='softmax')
  14.     cost = fluid.layers.cross_entropy(input=predict, label=label)
  15.     avg_cost = fluid.layers.mean(cost)
  16. prog_clip = prog.clone()
  17. avg_cost_clip = prog_clip.block(0).var(avg_cost.name)
  18. p_g = fluid.backward.append_backward(loss=avg_cost)
  19. p_g_clip = fluid.backward.append_backward(loss=avg_cost_clip)
  20. with fluid.program_guard(main_program=prog_clip, startup_program=startup_program):
  21.     fluid.clip.set_gradient_clip(
  22.         fluid.clip.GradientClipByNorm(clip_norm=2.0))
  23.     p_g_clip = fluid.clip.append_gradient_clip_ops(p_g_clip)
  24. grad_list = [elem[1] for elem in p_g]
  25. grad_clip_list = [elem[1] for elem in p_g_clip]
  26. train_reader = paddle.batch(
  27.     paddle.reader.shuffle(
  28.         paddle.dataset.mnist.train(), buf_size=8192),
  29.     batch_size=128)
  30.  
  31. exe = fluid.Executor(place)
  32. feeder = fluid.DataFeeder(feed_list=[image, label], place=place)
  33. exe.run(startup_program)
  34.  
  35. count = 0
  36. for data in train_reader():
  37.     count += 1
  38.     print("count:%s" % count)
  39.     if count > 5:
  40.        break
  41.     out = exe.run(prog, feed=feeder.feed(data), fetch_list=grad_list)
  42.     out_clip = exe.run(prog_clip,
  43.                        feed=feeder.feed(data),
  44.                        fetch_list=grad_clip_list)