Env : Azure Databricks Cluster : 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12) I have pandas_udf, its working for 4 rows, but I tried with more ...

my assignment is to store the following into an array type column: when I run that cmd, I got this error: TypeError: anomaly_detections() takes 1 ...

I'm trying to get the country name with latitude and longitude as input, so I used the Nominatim API and when I pass as a UDF it works, but when I try ...

So suppose I have a big spark dataframe .I dont know how many columns. (the solution has to be in pyspark using pandas udf. Not a different approach) ...

I am trying to return a StructField from a Pandas UDF in Pyspark used with aggregation with the following function signature: But it turns out that ...

I want to use pandas_udf in Pyspark for certain transformations and calculations of column. And it seems that pandas udf can't be written exactly as n ...

I have a created a geopandas dataframe with 50 million records which contain Latitude Longitude in CRS 3857 and I want to convert to 4326. Since the d ...

I have a dataframe df with the column sld of type string which includes some consecutive characters with no space/delimiter. One of the libraries that ...

I have a piece of code that I want to translate into a Pandas UDF in PySpark but I'm having a bit of trouble understanding whether or not you can use ...

I'm trying to create a column of standardized (z-score) of a column x on a Spark dataframe, but am missing something because none of it is working. H ...

I don't know if this question has been covered earlier, but here it goes - I have a notebook that I can run manually using the 'Run' button in the not ...

I have to divide a set of columns in a pyspark.sql.dataframe by their respective column average but I am not able to find an correct way to do it. Bel ...

I do try to compute .dot product between 2 columns of a give dataframe, SparseVectors has this ability in spark already so I try to execute this in ...

I'm trying to parallelize the training of multiple time-series using Spark on Azure Databricks. Other than training, I would like to log metrics and m ...

I do have an UDF that is slow for large dataset and I try to improve execution time and scalability by leveraging pandas_udfs and all searching and of ...