Unix timestamps in Spark and PySpark
Spark stores TIMESTAMP values as microseconds since 1970 UTC. Since Spark 3.1, SQL has direct functions for each unit.
Get the current Unix timestamp
-- Spark SQL
SELECT unix_seconds(current_timestamp()), unix_millis(current_timestamp());
# PySpark
from pyspark.sql import functions as F
df = df.withColumn('now_s', F.unix_timestamp())
Convert a Unix timestamp to a date
-- Spark SQL 3.1+
SELECT timestamp_seconds(1700000000); -- 2023-11-14 22:13:20 (shown in the session zone)
SELECT timestamp_millis(1700000000123);
# PySpark
df = df.withColumn('ts', F.timestamp_seconds('epoch_s'))
df = df.withColumn('ts', F.expr('timestamp_millis(epoch_ms)'))
Convert a date to a Unix timestamp
SELECT unix_seconds(TIMESTAMP '2026-11-14 10:00:00 UTC'); -- 1794650400
SELECT unix_millis(ts) FROM events;
df = df.withColumn('epoch_ms', F.expr('unix_millis(ts)'))
Timezones
spark.conf.set('spark.sql.session.timeZone', 'UTC') # how timestamps are shown and parsed
-- Show a UTC timestamp as wall-clock time in another zone
SELECT from_utc_timestamp(timestamp_seconds(1700000000), 'Asia/Kolkata');
Common mistakes
- Strings without a zone are parsed in
spark.sql.session.timeZone, which defaults to the JVM's zone. Set it to UTC on every cluster. from_unixtime()returns a string, not a timestamp. Prefertimestamp_seconds().CAST(epoch_ms AS TIMESTAMP)treats the number as seconds. Divide by 1000 first, or usetimestamp_millis.
Complete example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.config('spark.sql.session.timeZone', 'UTC').getOrCreate()
df = spark.createDataFrame([(1700000000,), (1794650400,)], ['epoch_s'])
df = (df.withColumn('ts', F.timestamp_seconds('epoch_s'))
.withColumn('epoch_ms', F.expr('unix_millis(ts)')))
df.show(truncate=False)
Check any value with the epoch converter, or convert a whole column with the batch converter. All examples use 1700000000 (2023-11-14 22:13:20 UTC) and 1794650400 (2026-11-14 10:00:00 UTC).