如何在python中展平RDD？

Question

I have a dataset of spam msgs and it has this datatype:

pyspark.rdd.PipelinedRDD

When I do spams.take(3) , I get:

[["Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's"], ['WINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.'], ['Had your mobile 11 months or more? UR entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030']]

As you can see it has brackets within to separate each element within the list. How can I get rid of those brackets? I tried many ways of Flattening it but none seems to work.

Answer 1

You can use flatMap method of rdd. It allow you to generate multiple rows from one row.

spams.flatMap(lambda x:x).take(3)

Answer 2

Since your question is unclear whether you want to remove brackets after collecting in list or before collecting and other users already answered for after , I will answer for while data is still a rdd. It is pretty straight forward,

spams = spams.map(lambda x:x[0])
print spams.take(3)

This will remove the inner "brackets".

Answer 3

These lines of code will help.

    >>> msg = [["Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 0
8452810075over18's"],
...  ['WINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Valid
12 hours only.'],
...  ['Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on
08002986030']]
>>> msg = [x[0] for x in msg]
>>> msg
["Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075o
ver18's", 'WINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Va
lid 12 hours only.', 'Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Upd
ate Co FREE on 08002986030']

Answer 4

Try a for loop, "data" is the list you get back from spam.take(3).

mylist = []
for entry in data:
  print(entry)
  for e in entry:
    mylist.append(e)
print(mylist)