2

I am using Jupyter notebook for running Spark. My problem arises when I am trying to register a UDF from my custom imported jar.

This is how I create th UDF in my custom jar:

package com.udf;
import org.apache.spark.sql.api.java.UDF1;
  
public class TestUDF implements UDF1<String,String> {

   public String call(String arg) throws Exception {
      return doSomehing(arg);
}
...

then I try to import it like this in Jupyter notebook

val spark = SparkSession.builder
.master("yarn")
.appName("Spark SQL")
.config("spark.jars", "/user/.../test.udf.jar")
.getOrCreate()

or like this

spark.sparkContext.addJar("/user/.../test.udf.jar")

Not sure if these imports work fine but there is no error message at least. Then trying to register my udf like this

spark.udf.register("myUDF", TestUDF.call)

I get an error message:

not found: value TestUDF

(Tried some other names but also not found)

This approach seems legit. But I couldn't find explanations about both importing jar and accessing udf from it. Am I missing something important? Could anyone help me with this?

Edit:

Maybe TestUDF should be explicitly imported like this

import com.udf.TestUDF

but this import attempt returns an error: object udf is not a member of package com

and registering

spark.udf.register("myUDF", new TestUDF(), StringType)

returns not found: type TestUDF

mamonu
  • 392
  • 3
  • 19
humme1
  • 367
  • 1
  • 4
  • 13
  • Have you tried : `spark.udf.register("myUDF", new TestUDF(), StringType)`? – blackbishop Jan 28 '21 at 17:53
  • I've seen reply here https://stackoverflow.com/questions/54771895/how-to-register-the-java-spark-udf-in-spark-shell but it's not checked as a solution and also there's an import from console not Jupyter – humme1 Jan 28 '21 at 18:26
  • @blackbishop error message is "not fount: type TestUDF " – humme1 Jan 28 '21 at 19:00

1 Answers1

1

Try first importing your jar file by using sparkContext

spark.sparkContext.addPyFile("yourjarfilepath")

then

spark.udf.registerJavaFunction("UDF1","com.udf.UDF1",StringType())
thalearningmenace
  • 21
  • 4