How to do direct bigquery subscription? Getting 400 error invalid argument

Google Cloud Data Analytics
,
1

https://gist.github.com/donbowman/5ea8f8d8017493cbfa3a9e4f6e736bcc shows all the details. I have also posted on stackoverflow: https://stackoverflow.com/questions/73767750/how-do-i-create-a-bigquery-pubsub-direct-in-gcp-i-get-an-error-failed-to-create

I have created a schema (initially in avro, but then in protobuf).

I have created a schema using this, and a topic using it.

I have then created a bigquery table using the schema.

When I create a subscription of type bigquery table, i get a 400 error, invalid argument.I have manually verified all the fields vs the protobuf. I cannot see anything wrong, and, there is no hint in the error message. The message sent is:

{"ackDeadlineSeconds": 900, "bigqueryConfig": {"dropUnknownFields": true, "table": "agilicus:checkme.httpobs", "useTopicSchema": true, "writeMetadata": true}, "name": "projects/agilicus/subscriptions/check-me.httpobs", "topic": "projects/agilicus/topics/check-me.httpobs"}

but all i get is an error each time.

If i purposely mispell the table name, there is no change in the error, so i believe the issue is with pubsub topic + schema somehow.

Does any one have a suggestion or example? I am quite stuck.

I have also tried using a python file to create the subscription: same effect.

2

At the moment, logical types are not supported with bigquery subscriptions. The documentation is being updated to reflect this fact and the support por logical types is being added, expect it to be ready by the end of Q3. At the moment the TIMESTAMP logical type would map as INT64.

If you want to be properly updated on when this change will be available, you can visit issue tracker.

3

in my schema, i put:

uint64 publish_time = 4
[ (gen_bq_schema.bigquery) = { require: true, type_override: ‘TIMESTAMP’ } ] //STRIP
;

as per the example in https://github.com/GoogleCloudPlatform/protoc-gen-bq-schema

when i generate this for pubsub, that becomes uint64 publish_time. When it generates for BQ, it becomes Timestamp.

4

As I previously mentioned, pub/sub at the moment doesn’t support the TIMESTAMP logical type, this will be accessible by the end of this month.

5

Hi Don

Do you know how to repeat the values in pubsub schema for BQ subscription?

6

pubsub message has list of json objects