freezing at a Vertex AI training job

Google Cloud Cloud Foundations & Onboarding
1

Hi,

I am trying to implement custom model training on Vertex AI.

Every time I pushed custom training job to Vertex AI, I could meet the freezing issue in the middle of the process.

The issue occurs in,


message: "Generating new fontManager, this may take some time..."
pathname: "/opt/conda/lib/python3.7/site-packages/matplotlib/font_manager.py"

I tried to remove matplotlib package either, but it still tries to setup fontManager during the process.

Is there anyone who’ve been facing a same issue or have solved before?
Thanks,

2

You can try using a non-interactive matplotlib backend like Agg especially if you use the plot function. Here’s the official documentation. To select a backend, you may refer here. If you can, you may also share code snippets for easier reproducing and debugging.