The Issue
If you are trying to access workspace files or files located in a repository folder (as described here) from a shared cluster in Databricks, you might run into the following error:
java.lang.SecurityException: User does not have permission SELECT on any file
The underlying problem has to do with the restrictions you are facing when using shared clusters in Databricks.
If you want to access Unity Catalog with a cluster the only two options regarding “Access Mode” currently are "Shared" or “Single User” though. If the latter is out of the question, as often happens in projects that i am participating in, then you will not be able to access workspace files from the only cluster option that is left.
The Solution
One way of dealing with this problem is to use Databricks Python SDK.
(1) Install the SDK on your Cluster
First you need to install the SDK on your current cluster. There are numerous ways to install libraries on your cluster, quick one is to add this line of code in a notebook cell of its own: