Hey GCP community,
We are using Cloud composer and using Dynamic Dag generation concept , but we storing the dag configuration has airflow variables and it is used for the dag creation.
Recently found that in the airflow documentation it is not good and it will create top level code and which can slow down parsing time
https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html
They suggest to use environmental variable to store the configuration.
I am little confused .
So , should need to use composer variable to store this configuration.
And i used to composer variable to store single key value pair like env: prod etc
These json configuration is around 1000 lines.how can store it in the composer variable
or this environmental variable is something else .
Can someone pls advice on this scenerio