BQ Loader - Script Error

Google Cloud Data Analytics
1

Dear All,

I try to implement a shell script in order to load a csv file into a dedicated table on BQ but after some execution I received the following error message:

Upload complete.Traceback (most recent call last):
  File "/appl/sw/google-cloud-sdk/platform/bq/bq.py", line 5837, in <module>
    appcommands.Run()
  File "/appl/sw/google-cloud-sdk/platform/bq/third_party/google/apputils/appcommands.py", line 770, in Run
    return app.run()
  File "/appl/sw/google-cloud-sdk/platform/bq/third_party/google/apputils/app.py", line 216, in run
    return _actual_start()
  File "/appl/sw/google-cloud-sdk/platform/bq/third_party/google/apputils/app.py", line 244, in _actual_start
    really_start()
  File "/appl/sw/google-cloud-sdk/platform/bq/third_party/google/apputils/appcommands.py", line 756, in _CommandsStart
    sys.exit(command.CommandRun(GetCommandArgv()))
  File "/appl/sw/google-cloud-sdk/platform/bq/third_party/google/apputils/appcommands.py", line 284, in CommandRun
    ret = self.Run(argv)
  File "/appl/sw/google-cloud-sdk/platform/bq/bq.py", line 1100, in Run
    return super(BigqueryCmd, self).Run(argv)
  File "/appl/sw/google-cloud-sdk/platform/bq/bq.py", line 1017, in Run
    return self.RunSafely(new_args, new_kwds)
  File "/appl/sw/google-cloud-sdk/platform/bq/bq.py", line 1107, in RunSafely
    return BigqueryCmd.ProcessError(e, name=self._command_name)
  File "/appl/sw/google-cloud-sdk/platform/bq/bq.py", line 1213, in ProcessError
    print(message)
IOError: [Errno 5] Input/output error

The strange behaviour is that the upload on BQ will be completed successfully.

Waiting on bqjob_r6f2eda560ac4437d_0000018d3ad5bede_1 … (3s) Current status: DONE

Please find below my script:

export PATH=/usr/lib64/python2.7:$PATH

echo $(date '+[%Y-%m-%d %H:%M:%S]') >> /appl/sw/tmp/bqload.log
echo "start shell bqload.sh \n" >> /appl/sw/tmp/bqload.log
echo $1 >> /appl/sw/tmp/bqload.log
echo $2 >> /appl/sw/tmp/bqload.log
echo $3 >> /appl/sw/tmp/bqload.log

"/appl/sw/google-cloud-sdk/bin/bq" show --schema "dataset-test:datamodel.$1" > /appl/sw/tmp/schema.csv.$3 2>>/appl/sw/tmp/bqload.log

if [ $? -eq 0 ]; then
   echo "Comando bq show terminato con successo" >> /appl/sw/tmp/bqload.log
else
   exit 1
fi

echo " esito file schema $? \n" >> /appl/sw/tmp/bqload.log

if [ -f $2 ]; then
   echo "Il file $2 esiste \n"  >> /appl/sw/tmp/bqload.log
else
   echo "il file non esiste \n"  >> /appl/sw/tmp/bqload.log
   exit 1
fi

"/appl/sw/google-cloud-sdk/bin/bq" load --field_delimiter="|" "dataset-test:datamodel.$1" "$2" /appl/sw/tmp/schema.csv.$3 2>>/appl/sw/tmp/bqload.log

if [ $? -eq 0 ]; then
   echo "Comando bq show terminato con successo" >> /appl/sw/tmp/bqload.log
else
   exit 1
fi

echo " esito load $? \n" >> /appl/sw/tmp/bqload.log
echo $(date '+[%Y-%m-%d %H:%M:%S]') >> /appl/sw/tmp/bqload.log
echo "end shell bqload.sh" >> /appl/sw/tmp/bqload.log

exit 0

More details about our environment below our Environment:

  • OS: RedHat 8.8
  • Python 2 (2.7)
  • Google SDK (Version 267.0.0)

could you please support me?

Best Regards
Simone

1 Like