When I save a PySpark DataFrame with saveAsTable in AWS EMR Studio, where does it get saved?

Question

I can save a dataframe using df.write.saveAsTable('tableName') and read the subsequent table with spark.table('tableName') but I'm not sure where the table is actually getting saved?

you can probably check the definition of your table to see where the data are stored. — Steven, Aug 24 '21 at 14:02

Sima · Answer 1 · 2022-01-20T18:03:07.840

1

It is stored under the default location of your database.

You can get the location by running the following spark sql query:

spark.sql("DESCRIBE TABLE EXTENDED tableName")

You can find the Location under the # Detailed Table Information section. Please find a sample output below:

edited Jan 20 '22 at 18:03

answered Jan 20 '22 at 17:38

Sima

11
2

How does the output looks like ? Maybe you can give an idea ? – Felix Jan 20 '22 at 17:47
I added a sample in the answer – Sima Jan 20 '22 at 18:03

When I save a PySpark DataFrame with saveAsTable in AWS EMR Studio, where does it get saved?

1 Answers1