I am trying to copy data from one mySql table to another mySql table, but while fetching data only 25 rows are pass to another flow(to load data to another table) instead of full data.
Below are my configurations,
Flow1: to fetch data from 1st table
flow2 : to load data to 2nd table
After testing flow1 it is in succeeded state always but only 25 rows are inserted. Can anyone suggest here how to fetch full data.
Hi @np1
Spookly enough we were just looking at this yesterday!
On your mapping step then you can set the ListEntitiesPageSize, so you can load in many more rows. Depending on your payload size you maybe able to get say 500 rows at a time. (There is a connector payload limit of 8M according to https://cloud.google.com/application-integration/docs/quotas#data-processing-limits)
If you need more rows you will really need to loop and when you call the first ‘page’ of data consume a returned variable called listEntitiesPageToken. This then has to be fed into your next loop iteration to get say the 2nd page of data.
More into on it can be found here:
https://cloud.google.com/application-integration/docs/configure-connectors-task#config-prop
I havent been able to test the latter on Postgres, as it doesn’t appear to be returning a ‘listEntitiesPageToken’, but hopefully mySql is better.
Hi @np1
Pls take a look at this thread for fetching large payload response from a connector: https://www.googlecloudcommunity.com/gc/Integration-Services/API-Pagination-in-Application-Integration-Best-Practice/m-p/631385/highlight/true#M489
Lmk if this answers your question
Hi @np1
I was having trouble getting the Nextpagetoken back so raised a case with google. It looks like you have to both map a pagesize and a sort column to get the token back (listEntitiesPageSize, listEntitiesSortByColumns-[NB String-Array] ). I have managed to do this and get a token back, which I then fed into a 2nd connector call (with also pagesize and sort column mapped) and it then returned the subsequent 25 rows.
Not sure why you have to specify a pagesize to get the token back, instead of it just picking up the default of 25!
It also looks like if you have a primary key on the table you dont need the sort definition (but I havent tested that).
Good luck
Hey @Graham-Vosper , I’m having the same issue as above.
I manage to get a token back when the only value I add islistEntitiesPageSize but I continue to get the same error as @np1 whenever I try to set the listEntitiesSortByColumns value to anything or if I try to set the value of listEntitiesPageToken to the listEntitiesNextPageToken.
Do you know if there is a specfiic format that should be used for the sort or page token columns? I noticed that listEntitiesSortByColumns should be a string array and i am entering it in the same format as below but continue to receive an error:
Maybe it could be down to that?
Hi @mfers
On the listEntitiesSortByColumns then what I did was to create a variable of type ‘String Array’ and then include the database column name as the Default Value as shown here:
(col_int is the name of my database column, note it doesnt allow you to have multiple sort columns)
On the question of the next page token mapping then it should be a case of mapping the ‘NextPageToken’ from your first call to the ‘PageToken’ of the next. For my dummy integration I had 2 connector calls, one to get the first 25 and another to get the next 25 as shown here.
The mapping stage prior to the ‘next 25’ call had:
This simple test worked fine for me to prove the concept. If I wanted to build something production like then I suspect I would need to have something that say reads the first 500 rows, and if the results are non-blank it would call the same integration to process the next 500 via an API Trigger call passing the next-page token across.
