Как заменить пробелы на Null, если у меня есть пробелы в нескольких столбцах.

Input Dataset which i have
+---+-----++----+
| Id|col_1|col_2|
+---+-----+-----+
|  0|104  |     |
|  1|     |     |
+---+-----+-----+
import org.apache.spark.sql.functions._

val test = df.withColumn("col_1","col_2", regexp_replace(df("col_1","col_1"), "^\\s*", lit(Null)))
test.filter("col_1,col_2 is null").show()

Выходной набор данных:

+---+-----++----+
| Id|col_1|col_2|
+---+-----+-----+
|  0|104  | Null|
|  1|Null | Null|
+---+-----+-----+
0
Praveen Saini 30 Май 2019 в 14:59

2 ответа

Лучший ответ

Используйте один withColumn для каждого столбца:

import org.apache.spark.sql.functions._
val df = List(("0", "104", "    "), ("1", " ", "")).toDF("Id","col_1", "col_2")

val test = df
  .withColumn("col_1", when(regexp_replace (col("col_1"), "\\s+", "") === "", null).otherwise(col("col_1")))
  .withColumn("col_2", when(regexp_replace (col("col_2"), "\\s+", "") === "", null).otherwise(col("col_2")))
  .show

Результат

+---+-----+-----+
| Id|col_1|col_2|
+---+-----+-----+
|  0|  104| null|
|  1| null| null|
+---+-----+-----+
1
Pablo López Gallego 31 Май 2019 в 08:09

Привет, Вы можете сделать так:

scala> val someDFWithName = Seq((1, "anurag", ""), (5, "", "")).toDF("id", "name", "age")
someDFWithName: org.apache.spark.sql.DataFrame = [id: int, name: string ... 1 more field]

scala> someDFWithName.show
+---+------+---+
| id|  name|age|
+---+------+---+
|  1|anurag|   |
|  5|      |   |
+---+------+---+
scala> someDFWithName.na.replace(Seq("name","age"),Map(""-> null)).show
+---+------+----+
| id|  name| age|
+---+------+----+
|  1|anurag|null|
|  5|  null|null|
+---+------+----+

Или попробуйте это также:

scala> someDFWithName.withColumn("Name", when(col("Name") === "", null).otherwise(col("Name"))).withColumn("Age", when(col("Age") === "", null).otherwise(col("Age"))).show
+---+------+----+
| id|  name| age|
+---+------+----+
|  1|anurag|null|
|  5|  null|null|
+---+------+----+

Или для более чем одного пробного пространства попробуйте это:

scala> val someDFWithName = Seq(("n", "a"), ( "", "n"), ("         ", ""), ("  ", "a"), ("   ",""), ("        ","   "), ("c"," ")).toDF("name", "place")
someDFWithName: org.apache.spark.sql.DataFrame = [name: string, place: string]

scala> someDFWithName.withColumn("Name", when(regexp_replace(col("name"),"\\s+","") === "", null).otherwise(col("Name"))).withColumn("Place", when(regexp_replace(col("place"),"\\s+","") === "", null).otherwise(col("place"))).show
+----+-----+
|Name|Place|
+----+-----+
|   n|    a|
|null|    n|
|null| null|
|null|    a|
|null| null|
|null| null|
|   c| null|
+----+-----+

Я надеюсь, что это поможет вам. Благодарность

1
Learner 30 Май 2019 в 19:17
56377905