PySpark DataFrame summary() for String and Numeric Columns

DP-600 By islever Editorial Last updated

Short answer

A Using df.summary() meets the goal of calculating summary statistics for all string and numeric columns.

Question

Note: This question is part of a series of questions that present the same scenario. Each question in the series contains a unique solution that might meet the stated goals. Some question sets might have more than one correct solution, while others might not have a correct solution. After you answer a question in this section, you will NOT be able to return to it. As a result, these questions will not appear in the review screen. You have a Fabric tenant that contains a new semantic model in OneLake. You use a Fabric notebook to read the data into a Spark DataFrame. You need to evaluate the data to calculate the min, max, mean, and standard deviation values for all the string and numeric columns. Solution: You use the following PySpark expression: df.summary() Does this meet the goal?
  1. Yes Correct Answer
  2. No

Key Takeaways

The question tests knowledge of PySpark's df.summary() function, specifically that it operates on both string and numeric columns, unlike pandas' default describe().

This page confirms that the PySpark df.summary() method correctly calculates summary statistics for both string and numeric columns. It establishes that the method meets the goal of evaluating data for min, max, mean, and standard deviation.

Common mistake: Choosing 'No' because mean and standard deviation are not applicable to string columns, failing to realize df.summary() still includes string columns and computes applicable statistics like min and max.

Expert Analysis

Why the Answer Is Correct

The df.summary method in PySpark is explicitly designed to compute summary statistics for both numeric and string columns. For numeric columns, it calculates count, mean, stddev, min, max, and optional percentiles. For string columns, it calculates count, min, and max, returning null for inapplicable statistics like mean and stddev. Therefore, using df.summary successfully meets the goal of evaluating the data across all string and numeric columns.

Why the Other Options Are Wrong

Option B suggests the goal is not met, which is incorrect. A common misconception is that df.summary ignores string columns entirely or fails if it cannot compute mean and stddev for them. While those specific statistical measures are undefined for strings, the method still processes string columns to find applicable statistics like min and max, fully satisfying the requirement to evaluate all column types.

Community Comment Notes

Several commenters correctly noted that df.summary works on both string and numeric columns, with one stating "In PySpark, use df.summary for summary statistics of both numeric and string columns". Others argued against it because "mean and standard deviation are not applicable to string data", but the function gracefully handles this by returning null for those metrics while still evaluating the string columns. Another commenter incorrectly claimed "df.summary.show is correct", but the calculation occurs within the summary method itself.

Official Reference

https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.summary.html

This link opens the official vendor documentation in a new tab.

Exam Strategy

Remember that PySpark's df.summary() and df.describe() process both string and numeric columns, unlike pandas which defaults to numeric only. Do not assume the function fails on string columns just because some statistics are undefined.

Community Votes

A
55%
B
45%

55% of anonymous learners picked answer A. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 11 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 11

  1. stilferx 👍 9
    IMHO, A Example: df1 = spark.createDataFrame([(1, 10), (2, 10), (2, 15)], schema = ['fruit_id', 'amount']) df1.summary() summary fruit_id amount count 3 3 mean 1.6666666666666667 11.666666666666666 stddev 0.5773502691896257 2.886751345948129 min 1 10 25% 1 10 50% 2 10 75% 2 15 max 2 15
  2. 282b85d 👍 4 Answered B
    while df.summary() does provide valuable information for numeric columns, it does not fully meet the goal of evaluating both string and numeric columns with the required statistical measures. Use df.summary() and df.agg() to cover numeric columns, and additional custom aggregations for string columns.
  3. SamuComqi 👍 4 Answered A
    Also df.describe() is a valid solution. Sources: summary --> https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.summary.html describe --> https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.describe.html
  4. 7d97b62 👍 3 Answered A
    In pandas, use df.describe() for summary statistics of numeric columns. In PySpark, use df.summary() for summary statistics of both numeric and string columns in a distributed computing environment.
  5. Pegooli 👍 2 Answered B
    Using df.summary() in PySpark will provide summary statistics, including min, max, mean, and standard deviation for all numeric columns. However, it will not provide these statistics for string columns since summary statistics like min, max, mean, and standard deviation are not applicable to string data.
Show 6 more comments
  1. 2fe10ed 👍 1 Answered A
    https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.summary.html#pyspark.sql.DataFrame.summary
  2. b01d700 👍 1 Answered B
    The correct PySpark expression to calculate min, max, mean, and standard deviation for both numeric and STRING columns is: df.describe()
  3. Lotusss 👍 1 Answered B
    df.summary() alone wont cut it. df.summary().show() is correct. So anwser is B
  4. XiltroX 👍 1
    df.summary() is the only option where you can get MIX, MAX and AVG
  5. slu239 👍 1 Answered B
    Not meet the goal because it has to be df.summary().show()
  6. 6d1de25 👍 1 Answered A
    Correct

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Fabric Deployment Pipeline Folder Structure Behavior

S-grade

Accessing OneLake Shortcuts in Fabric Notebooks

S-grade

Direct Lake Fallback to DirectQuery with Row-Level Security

S-grade

Displaying All Rows in Power BI Python Visuals

S-grade

Practice All DP-600 Questions

Access 115 questions with complete answers and detailed explanations.

View Full DP-600 Practice Test →

← Back to DP-600 Study Guide