The Databricks cluster is using UTC as the default timezone. So when you run some time-related code, the displayed time is not the local time, which is not ideal. In this post, I want to share how to change the timezone setting for Databricks cluster.

Change the system timezone#

In the notebook cell, run the following to check what Linux system Databricks is using:

It shows the underlying system is Ubuntu. On Ubuntu, we can use timedatectl command line tool to change the timezone.

List timezones:

%sh
timedatectl list-timezones

Pick a timezone we want:

%sh
timedatectl set-timezone Asia/Shanghai

Note the above command does not impact the timezone setting of spark, since spark has already been started.

ref:

• change timezone Ubuntu: https://askubuntu.com/a/594186/768311

For the current session#

If you only want to set the timezone for current spark session, just run the following statement:

spark.conf.set('spark.sql.session.timeZone', 'Asia/Shanghai')

The explanation for spark.sql.session.timeZone:

The ID of session local timezone in the format of either region-based zone IDs or zone offsets. Region IDs must have the form ‘area/city’, such as ‘America/Los_Angeles’. Zone offsets must be in the format ‘(+|-)HH’, ‘(+|-)HH: mm’ or ‘(+|-)HH:mm:ss’, e.g ‘-08’, ‘+01:00’ or ‘-13:33:33’. Also ‘UTC’ and ‘Z’ are supported as aliases of ‘+00:00’. Other short names are not recommended to use because they can be ambiguous.

Then you can run display(spark.sql("select current_timezone()")) in Databricks notebook cell to verify the change. However, if you create a new notebook using the same cluster, the timezone setting does not persist.

ref:

• spark configuration: https://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration
• set default timezone databricks: https://stackoverflow.com/a/68268838/6064933
• use different timezone for spark session: https://stackoverflow.com/a/68957356/6064933

In the cluster config#

In the Advanced options section of cluster setting page, under Spark tab, you can add the following config:

spark.sql.session.timeZone Asia/Shanghai

This will make sure every notebooks that attached to this cluster has the correct timezone setup.

databricks cluster init script#

You can also set up the timezone in init script for the cluster. Just add someting like the following to the cluster init script:

timedatectl set-timezone Asia/Shanghai

Precedence of different settings#

The precedence of setting up timezone of the above three methods:

setting in notebook session > cluster config > init script

ref:

• what are init script: https://learn.microsoft.com/en-us/azure/databricks/init-scripts/
• cluster-scope init script: https://learn.microsoft.com/en-us/azure/databricks/init-scripts/cluster-scoped