PySpark DataFrame summary() for String and Numeric Columns

Answer Correct answer: A — Using df.summary() meets the goal of calculating summary statistics for all string and numeric columns.

Note: This question is part of a series of questions that present the same scenario. Each question in the series contains a unique solution that might meet the stated goals. Some question sets might have more than one correct solution, while others might not have a correct solution. After you answer a question in this section, you will NOT be able to return to it. As a result, these questions will not appear in the review screen. You have a Fabric tenant that contains a new semantic model in OneLake. You use a Fabric notebook to read the data into a Spark DataFrame. You need to evaluate the data to calculate the min, max, mean, and standard deviation values for all the string and numeric columns. Solution: You use the following PySpark expression: df.summary() Does this meet the goal?

Yes Correct Answer
No

pyspark df.summary dataframe summary statistics pyspark string numeric columns fabric notebook pyspark

Community Votes

55%

45%

55% of anonymous learners picked answer A. Votes are pick records left by other test-takers — they are not the verified answer.

Community Insight

The question tests knowledge of PySpark's df.summary() function, specifically that it operates on both string and numeric columns, unlike pandas' default describe().

This page confirms that the PySpark df.summary() method correctly calculates summary statistics for both string and numeric columns. It establishes that the method meets the goal of evaluating data for min, max, mean, and standard deviation.

Choosing 'No' because mean and standard deviation are not applicable to string columns, failing to realize df.summary() still includes string columns and computes applicable statistics like min and max.

Community Discussion (12 comments)

stilferx 👍 9

IMHO, A Example: df1 = spark.createDataFrame([(1, 10), (2, 10), (2, 15)], schema = ['fruit_id', 'amount']) df1.summary() summary fruit_id amount count 3 3 mean 1.6666666666666667 11.666666666666666 stddev 0.5773502691896257 2.886751345948129 min 1 10 25% 1 10 50% 2 10 75% 2 15 max 2 15

Lotusss 👍 1 Selected: B

df.summary() alone wont cut it. df.summary().show() is correct. So anwser is B

b01d700 👍 1 Selected: B

The correct PySpark expression to calculate min, max, mean, and standard deviation for both numeric and STRING columns is: df.describe()

slu239 👍 1 Selected: B

Not meet the goal because it has to be df.summary().show()

2fe10ed 👍 1 Selected: A

https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.summary.html#pyspark.sql.DataFrame.summary

Pegooli 👍 2 Selected: B

Using df.summary() in PySpark will provide summary statistics, including min, max, mean, and standard deviation for all numeric columns. However, it will not provide these statistics for string columns since summary statistics like min, max, mean, and standard deviation are not applicable to string data.

6d1de25 👍 1 Selected: A

Correct

7d97b62 👍 3 Selected: A

In pandas, use df.describe() for summary statistics of numeric columns. In PySpark, use df.summary() for summary statistics of both numeric and string columns in a distributed computing environment.

282b85d 👍 4 Selected: B

while df.summary() does provide valuable information for numeric columns, it does not fully meet the goal of evaluating both string and numeric columns with the required statistical measures. Use df.summary() and df.agg() to cover numeric columns, and additional custom aggregations for string columns.

XiltroX 👍 1

df.summary() is the only option where you can get MIX, MAX and AVG

SamuComqi 👍 4 Selected: A

Also df.describe() is a valid solution. Sources: summary --> https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.summary.html describe --> https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.describe.html

Momoanwar 👍 2 Selected: A

Correct

Comments & Corrections

No comments yet — spotted an error or have a note? Share it below.

Expert Analysis

Why the Answer Is Correct

The df.summary method in PySpark is explicitly designed to compute summary statistics for both numeric and string columns. For numeric columns, it calculates count, mean, stddev, min, max, and optional percentiles. For string columns, it calculates count, min, and max, returning null for inapplicable statistics like mean and stddev. Therefore, using df.summary successfully meets the goal of evaluating the data across all string and numeric columns.

Why the Other Options Are Wrong

Option B suggests the goal is not met, which is incorrect. A common misconception is that df.summary ignores string columns entirely or fails if it cannot compute mean and stddev for them. While those specific statistical measures are undefined for strings, the method still processes string columns to find applicable statistics like min and max, fully satisfying the requirement to evaluate all column types.

Community Comment Notes

Several commenters correctly noted that df.summary works on both string and numeric columns, with one stating "In PySpark, use df.summary for summary statistics of both numeric and string columns". Others argued against it because "mean and standard deviation are not applicable to string data", but the function gracefully handles this by returning null for those metrics while still evaluating the string columns. Another commenter incorrectly claimed "df.summary.show is correct", but the calculation occurs within the summary method itself.

Official Reference

https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.summary.html

Exam Strategy

Remember that PySpark's df.summary() and df.describe() process both string and numeric columns, unlike pandas which defaults to numeric only. Do not assume the function fails on string columns just because some statistics are undefined.

Related Analysis

Practice All DP-600 Questions

Access 115 questions with complete answers and detailed explanations.

View Full DP-600 Practice Test →

← Back to DP-600 Study Guide