How to get applicationId of Spark application deployed to YARN in Scala?

Question

I'm using the following Scala code (as a custom spark-submit wrapper) to submit a Spark application to a YARN cluster:

val result = Seq(spark_submit_script_here).!!

All I have at the time of submission is spark-submit and the Spark application's jar (no SparkContext). I'd like to capture applicationId from result, but it's empty.

I can see in my command line output the applicationId and rest of the Yarn messages:

INFO yarn.Client: Application report for application_1450268755662_0110

How can I read it within code and get the applicationId ?

I think that yarn.Client is somehow getting the SparkContext.applicationId - you could do the same. — Markon, Jan 04 '16 at 09:53
Possible duplicate of [spark Yarn mode how to get applicationId from spark-submit](https://stackoverflow.com/questions/44209462/spark-yarn-mode-how-to-get-applicationid-from-spark-submit) — Rahul Sharma, Jun 05 '17 at 18:09

score 8 · Accepted Answer · answered Jan 04 '16 at 10:31

As stated in the Spark issue 5439, you could either use SparkContext.applicationId or parse the stderr output. Now, as you are wrapping the spark-submit command with your own script/object, I would say you need to read the stderr and get the application id.

score 7 · Answer 2 · answered Feb 04 '18 at 19:03

If you are submitting the job via Python, then this is how you can get the yarn application id:

    cmd_list = [{
            'cmd': '/usr/bin/spark-submit --name %s --master yarn --deploy-mode cluster '
                   '--executor-memory %s --executor-cores %s --num-executors %s '
                   '--class %s %s %s'
                   % (
                       app_name,
                       config.SJ_EXECUTOR_MEMORY,
                       config.SJ_EXECUTOR_CORES,
                       config.SJ_NUM_OF_EXECUTORS,
                       config.PRODUCT_SNAPSHOT_SKU_PRESTO_CLASS,
                       config.SPARK_JAR_LOCATION,
                       config.SPARK_LOGGING_ENABLED
                   ),
            'cwd': config.WORK_DIR
        }]
cmd_output = subprocess.run(cmd_obj['cmd'], shell=True, check=True, cwd=cwd, stderr=subprocess.PIPE)
cmd_output = cmd_output.stderr.decode("utf-8")
yarn_application_ids = re.findall(r"application_\d{13}_\d{4}", cmd_output)
                if len(yarn_application_ids):
                    yarn_application_id = yarn_application_ids[0]
                    yarn_command = "yarn logs -applicationId " + yarn_application_id

score 6 · Answer 3 · answered Jul 26 '18 at 14:12

6

Use the spark context to get application info.

sc.getConf.getAppId 
res7: String = application_1532296406128_16555

answered Jul 26 '18 at 14:12

mmopu

91
1
6

score 0 · Answer 4 · answered Aug 25 '21 at 01:43

as Rajiv's answer , the regex 'application_\d{13}_\d{4}' is not correct

actualy, the job id will increase greater than 9999, so the regex of application_\d{13}_\d{4,} will just working

and the java code

   public static final String APPLICATION_REGEX="application_\\d+_\\d{4,}+";

   /**
     * get yarn application id list
     * @param log   log content
     * @return app id list
     */
    public static List<String> getAppIds(String log) {
        List<String> appIds = new ArrayList<>();
        Matcher matcher = APPLICATION_REGEX.matcher(log);
        while (matcher.find()) {
            String appId = matcher.group();
            if(!appIds.contains(appId)){
                appIds.add(appId);
            }
        }
        return appIds;
    }

How to get applicationId of Spark application deployed to YARN in Scala?

4 Answers4

Linked