Let’s say my task is embarrassingly parallel and I have set parallelism to 1,000. I can run my program twice as fast on a n1-standard-32 than a n1-standard-16, it doesn’t really matter to me which one I use, I just want to have the largest of cores in total across of machines available. I am using batch jobs.
For example:
- there are 1,000 n1-standard-16 and only 495 n1-standard-32 available, I rather use the 16 core machines
- there are 200 n1-standard-16 but 105 n1-standard-32, I would rather use the 32 core machines
The issue is I want to use spot instances, but I don’t have a way of telling the available pool of spot instances for each machine type so I can’t make an informed decision on this. Is there a way to get this information?