Scalar User Defined Functions (UDFs)

Description

User-Defined Functions (UDFs) are user-programmable routines that act on one row. This documentation lists the classes that are required for creating and registering UDFs. It also contains examples that demonstrate how to define and register UDFs and invoke them in Spark SQL.

UserDefinedFunction

To define the properties of a user-defined function, the user can use some of the methods defined in this class.

  • asNonNullable(): UserDefinedFunction

    Updates UserDefinedFunction to non-nullable.

  • asNondeterministic(): UserDefinedFunction

    Updates UserDefinedFunction to nondeterministic.

  • withName(name: String): UserDefinedFunction

    Updates UserDefinedFunction with a given name.

Examples

  1. import org.apache.spark.sql.SparkSession
  2. import org.apache.spark.sql.functions.udf
  4. val spark = SparkSession
  5.   .builder()
  6.   .appName("Spark SQL UDF scalar example")
  7.   .getOrCreate()
  9. // Define and register a zero-argument non-deterministic UDF
  10. // UDF is deterministic by default, i.e. produces the same result for the same input.
  11. val random = udf(() => Math.random())
  12. spark.udf.register("random", random.asNondeterministic())
  13. spark.sql("SELECT random()").show()
  14. // +-------+
  15. // |UDF()  |
  16. // +-------+
  17. // |xxxxxxx|
  18. // +-------+
  20. // Define and register a one-argument UDF
  21. val plusOne = udf((x: Int) => x + 1)
  22. spark.udf.register("plusOne", plusOne)
  23. spark.sql("SELECT plusOne(5)").show()
  24. // +------+
  25. // |UDF(5)|
  26. // +------+
  27. // |     6|
  28. // +------+
  30. // Define a two-argument UDF and register it with Spark in one step
  31. spark.udf.register("strLenScala", (_: String).length + (_: Int))
  32. spark.sql("SELECT strLenScala('test', 1)").show()
  33. // +--------------------+
  34. // |strLenScala(test, 1)|
  35. // +--------------------+
  36. // |                   5|
  37. // +--------------------+
  39. // UDF in a WHERE clause
  40. spark.udf.register("oneArgFilter", (n: Int) => { n > 5 })
  41. spark.range(1, 10).createOrReplaceTempView("test")
  42. spark.sql("SELECT * FROM test WHERE oneArgFilter(id)").show()
  43. // +---+
  44. // | id|
  45. // +---+
  46. // |  6|
  47. // |  7|
  48. // |  8|
  49. // |  9|
  50. // +---+

Find full example code at “examples/src/main/scala/org/apache/spark/examples/sql/UserDefinedScalar.scala” in the Spark repo.

  1. import org.apache.spark.sql.*;
  2. import org.apache.spark.sql.api.java.UDF1;
  3. import org.apache.spark.sql.expressions.UserDefinedFunction;
  4. import static org.apache.spark.sql.functions.udf;
  5. import org.apache.spark.sql.types.DataTypes;
  7. SparkSession spark = SparkSession
  8.   .builder()
  9.   .appName("Java Spark SQL UDF scalar example")
  10.   .getOrCreate();
  12. // Define and register a zero-argument non-deterministic UDF
  13. // UDF is deterministic by default, i.e. produces the same result for the same input.
  14. UserDefinedFunction random = udf(
  15.   () -> Math.random(), DataTypes.DoubleType
  16. );
  17. random.asNondeterministic();
  18. spark.udf().register("random", random);
  19. spark.sql("SELECT random()").show();
  20. // +-------+
  21. // |UDF()  |
  22. // +-------+
  23. // |xxxxxxx|
  24. // +-------+
  26. // Define and register a one-argument UDF
  27. spark.udf().register("plusOne",
  28.   (UDF1<Integer, Integer>) x -> x + 1, DataTypes.IntegerType);
  29. spark.sql("SELECT plusOne(5)").show();
  30. // +----------+
  31. // |plusOne(5)|
  32. // +----------+
  33. // |         6|
  34. // +----------+
  36. // Define and register a two-argument UDF
  37. UserDefinedFunction strLen = udf(
  38.   (String s, Integer x) -> s.length() + x, DataTypes.IntegerType
  39. );
  40. spark.udf().register("strLen", strLen);
  41. spark.sql("SELECT strLen('test', 1)").show();
  42. // +------------+
  43. // |UDF(test, 1)|
  44. // +------------+
  45. // |           5|
  46. // +------------+
  48. // UDF in a WHERE clause
  49. spark.udf().register("oneArgFilter",
  50.   (UDF1<Long, Boolean>) x -> x > 5, DataTypes.BooleanType);
  51. spark.range(1, 10).createOrReplaceTempView("test");
  52. spark.sql("SELECT * FROM test WHERE oneArgFilter(id)").show();
  53. // +---+
  54. // | id|
  55. // +---+
  56. // |  6|
  57. // |  7|
  58. // |  8|
  59. // |  9|
  60. // +---+

Find full example code at “examples/src/main/java/org/apache/spark/examples/sql/JavaUserDefinedScalar.java” in the Spark repo.