pyspark.sql.functions.theta_union

pyspark.sql.functions.theta_union(col1, col2, lgNomEntries=None)

Merges two binary representations of Datasketches ThetaSketch objects, using a Datasketches Union object.

New in version 4.1.0.

Parameters

col1Column or column name

col2Column or column name

lgNomEntriesColumn or int, optional

The log-base-2 of nominal entries for the union operation (must be between 4 and 26, defaults to 12)

Returns

Column

The binary representation of the merged ThetaSketch.

See also

pyspark.sql.functions.theta_union_agg()

pyspark.sql.functions.theta_sketch_agg()

pyspark.sql.functions.theta_sketch_estimate()

Examples

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([(1,4),(2,5),(2,5),(3,6)], "struct<v1:int,v2:int>")
>>> df = df.agg(
...     sf.theta_sketch_agg("v1").alias("sketch1"),
...     sf.theta_sketch_agg("v2").alias("sketch2")
... )
>>> df.select(sf.theta_sketch_estimate(sf.theta_union(df.sketch1, "sketch2"))).show()
+--------------------------------------------------------+
|theta_sketch_estimate(theta_union(sketch1, sketch2, 12))|
+--------------------------------------------------------+
|                                                       6|
+--------------------------------------------------------+