Hi Joy,
Thank you for your detailed suggestions regarding the 404 errors on my sports-ingest-fanout service. Based on your feedback, I’ve conducted a thorough investigation and have an important update.
First, I can confirm I have followed all the steps you outlined for the original issue:
- The service URL is correct.
- The Ingress settings are confirmed to be --ingress=all, and the allUsers principal has the roles/run.invoker role.
- I ran the VPC-SC log query you provided. It returned zero results, suggesting VPC-SC is not the cause.
Following that, I’ve discovered and fixed several user-level configuration errors on our scheduler jobs (incorrect OIDC audience, wrong Content-Type headers). The good news is that for the sports-ingest-fanout service, this has resolved the connectivity issue. Jobs like delta-load-daily that target this service are now successfully reaching it. They currently return a 400 Bad Request, which we’ve traced in our application logs to an external API rate limit being hit. This is expected and proves the underlying 404 block for that service is gone.
However, this process has revealed a second, more persistent 404 Not Found error with a different scheduler job: trigger-odds-orchestrator.
The Unresolved Issue: trigger-odds-orchestrator
This job is designed to execute a Cloud Run Job (odds-ingest-orchestrator) by calling the run.googleapis.com admin API. It fails every time with a 404 Not Found, even though we have exhaustively verified the following:
- Resource Exists: The Cloud Run Job odds-ingest-orchestrator is confirmed to exist in the correct region (us-central1).
- API is Enabled: The Cloud Run Admin API is enabled for this project.
- Permissions are Correct: We granted the roles/run.invoker role to the job’s service account (odds-orchestrator-sa@…) and allowed ample time for IAM propagation.
- A “Clean Slate” Reset: To rule out any misconfiguration, we completely deleted and then programmatically recreated both the scheduler job and the Cloud Run Job. The 404 error immediately persisted on the brand-new resources.
Conclusive Finding: Pub/Sub vs. HTTP
This is the most critical piece of evidence we’ve found:
- A job that WORKS: Our engineered-nightly scheduler job, which triggers an internal Pub/Sub topic, succeeds without error every time.
- Jobs that FAIL: Every scheduler job that uses an HTTP target (whether to our own service or the Cloud Run Jobs API) is blocked at the transport layer.
This implies the problem is not with Cloud Scheduler in general, but specifically with a platform-level block on its HTTP trigger mechanism in this project.
Given that we have now ruled out all user-configurable causes, including VPC-SC, can you please escalate this case for an internal investigation to identify and resolve this platform-level block.
Thank you in advance for your help!
Mark
Here is the debug response:
~/sports-pipeline (appthree-86b53)$ curl -i -X POST “$JOB_URI”
-H “Authorization: Bearer $AUTH_TOKEN”
-H “Content-Type: application/json”
-H “X-Return-Encrypted-Headers: request_and_response”
-d ‘{}’
HTTP/2 404
date: Sun, 13 Jul 2025 00:21:35 GMT
content-type: text/html; charset=UTF-8
server: ESF
content-length: 1643
x-xss-protection: 0
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
x-encrypted-debug-headers: AIlid+HfAe5S9kqyQMqMX3mH/Uewa7pbMmVxgC9u/vWX645kss/k5AzWf8fHdBfcVbiFRdVBLO43EKlFTr5USOUb7q7JGOwTHftr5R9exzTitIUzQVQbg+uaqShSwUiD52gHid293iiIEXGR9gnkeIN4KTnxjIX8kjZS4uS+1xkhQsO3WGf0ac72Jzpfoc9jDLAHP22SSAIMcZIqX5ATsICvw5i64lqDul4KqGl92sPgRm1nr0z9jcydODjDknYDQEgChm9NyCggeY4yUFEwJPGazCBQPHTIrOCqjYcHozLszLFnU6XiAPaLIIsZbgDS4zcPtlwwG26MlNKjvXdJNccqxFZ2p5jvbnxhPCfvUx3SGXyojcG4ciIDOr9pFv0iF0Ptmkqu3/hzfVW21KT1ZRLH6VPITURueN4IgQOtZHV3xtewG8mX5skF2hkbc81aU9V8VnXir429GPXdr3Too0bIuqe9htu4dXsbw4FZQmNfXMbsTLVQdR5QWxGxzyPPuii4nbjsEAOxzjb5d4tYecVuJcva+BW8YgjCybbGGJlJ5Bn8S8oGBYKPYAT0yHYWZPmTqt8BZ7F0kJMgBC6wMP279GKi9Or8uIdDhbKHw2gnMenJ2n1d4aOrCYnZ3Y2N0RHOzJWFI0fRM5fAqFAPzYXnBYrzfV+qwuXZ/cE2Ow/ul/P1nO+rFVUjdX7f1cdaIJTUY2l0y7f9I/Fcm4uQezw7eNCSfx8ShY4LiO8fvHic6amNbrv6Bgyz2JRignq4oUcmt1x4OiNYOEvrsWYDwxMC2W9Pr1JbdZHFd4lNkx4yB4VwpjwkY7ZnRpMbCUeCedBprS8Yoph5nhVQmNreBWFC0wP4z+G6Fb3WvcZs2wvJ4dcxUhzBTMqsiC0UQ8qGTIyS9Xx8CqVDpDfo/xFm6yXYEC3YPgLXw0RuFZ3o1S3dvDJwUijTIq1nEpuYkHGJpeC4FUQlJrVGfZ6hcNOiOQx4BvipXRpZ+VzSyGpXkdWCMIzTd/sCjmVtLuhlij8PnUCIPsvQyIey55lWfa0qztRNGNt7H0Xs9t0d7y3KUAeYE4Epte3oQOOSp4rraB2TfhbtPoMQmKFBu7Ei5WaHnETxsoKqPpMXZa83qJM87VBDkxWv6FpGJf4xO2CU3QLpdYRW0oRlBOhAOAzpARit+ELHDXdaXFwwMsNPABbwqb3zbU8ewPxFahKjeai1iKWgYKF83JTtJt/y/AE+62FZ4ded2hsNsotsmoWHAR/mdTIt2tNsJTUAVx+cpOEiSGSUFPKocOEbzNSrKegcY9NvAKZabg59PvXVnoRU6tQCfHtWFXndCaMLfYVrJimuzZzcQYnlP1aUClUjvk1bf8KdbMW7jy2EJ4YH3wIGVt5jnIqHA/dybjkH4t5ZzdSDlb8KFUnkgzYr+7t07Bo/zxNWIZ6miqdIej8UaftpQ/lBUNXaXaWd4yeSu6Da0wx0vTfztsoB9h8WHZ97HnNYefdBni9ORV5BJ+Om9iXkp7iYka9Xblqwg8W+WlPCKclMBovsjvZy5ChH0wM4Pt1r/pjUShE8UrvCfk+CD9cZLK79SwK53cOHgyysGdz8e2vJp65y6ZSMgRf71aLt14ZvLeXWLqaAmtIJWYh9LLL9Mui9hDbyDp+K/1DNKidTXxEVqhX1obb8EVAZ+q8xqR+Ynfc1ok3l4352N0hoFjR6gm/i/e5+n+k3yWTgQy06DChqmTRrJKhcmhusp7bYuSu/VDkCEf8f8kK3hb0oRJfSPpGvlpizuU3Yb9NjBsCXCiq0iCN67HmfdRqC5Lxi0QlOqgnYuvS6LKe0aqOJURm8JCYWSQQtYusb9gKvOZE9b9YGtnebAqhPYujrgiJJ6MtUhMTQxMu/uzxtWvfXUlhqnyYIlEdKVgr8FUqkh1c17SRhyCIJGugpcYMzJUVhOvl2vfgkgB4hilNKTLF1DfAKW1PXauOa7GSNlFNcGIgS8x3aMJIViLQCmTpbjPEAsCey6QFoIiFbpUx8f3A14kJK926fmNSgvObfOMlZ6gKT17D9dMeEnpENTycifKmJrKRXtFtRCN7W+NWL68ZkEk2pM6kfwSQIzCNPkhde+/Us3w0215SLlNbmBuSChaOuDky184WbYhsNDHlBuLH6SmmNgRo6S3pEGgegshUfNTG6ID0d36zqM0QH0jAnPt2jMigKkGKRx6yKkb0T3ZxEEUdDjnOAVd7Atc8e7zm0vaVMCGF3r+2bqknVyaXqfUAQGZmgjQEoMDFr2vXSXPw+belGomDt9Qfihx3U83B7eI6x/KJ8Xi84VotdPkdkpfPH/HzSfC20JHlzu+rB40tobSpSziAgEHWwyo94ctAOOAOEYhit+LLGLDgOO5agZoo1X5einJ30i5KdogWQTFMIYSED4gXZpmrnrdNKT3vDHY06QraacsdSdlJQ4EHeFcJCXQZ7Z1+yaBssN+EIUbHADHcSa4/op4gW+pxSM9urGDcEsGxkkyIRbQm+8MBdu9wBhhdpB9xwCSkXvrW0n58fpEePOSLRPeFLIEeaA0qDaBfCukFPZdH76NGl/2f9BjVuBeG0Mdk1CZvqpRTOiRB9MY4+WCXzIEI0VYVZN12P7Jggye5UP+WqooLyqwkRrK4V2O72IeXpRH+H+Gm99nJuPpuvv2UxkIPhge3/lsPJQvTnkaRgud5TdeUQaSmMCitXHBFK01ZmppGY20pBJZ7o2P0FOn/5nkkbgCuFVvRztlf419STUXLTumumPe3+iFSwbbes8swNv1rm+pyYHkmW8dx3xZ7eYl4LJw9Jod56D6ctXE2L7mf/v/cFLeyH1L1GdMZqio1ucGonE+rPhGqk2i4bHuqy1P85XqI6lpp+/I32gvpzmoK1jdUtzzYqapoK02ZCrQbWkjDnLne3evMEEvtJl+RznQJiiKwLPSSZbqfxVgLty/wiUJgA3mVIKj/44Mw1rrw6qhgf3WWgM4fmHnlKk1nhVO8kmZp6qhd9KYLnB0jLuh9vxyTjYM9ZiIQ1xL7TcnT60/YZLYzxZvpasx3/zI45FImgsswL7AwAHiItOgwD+J/QhsIJkLX3aesseq6cxDsWcEETRnsXLHBYfNwa+NvJ8GGhHFjFbZkdLYECbba0zMwumWkXaCI6h0vACX+B4KHZKiDtVMKKGFfk/B3h1FkUrhdjg0mXVIVXwvC227ZHseKD0UdCtuJWE10JU5GpjViH2mQS+26LtfvGXd7QjtsQPn/D2rFmVCx7jDVUGxgotdZC0IrKnFaTeRYQD+VoER6tRKwnZqV46RXp/Jfw7Q5W4seD4BXsMQ2ohKBdmRHkPJfeHsIo7v8US+BEWdf0P/TdnEh3stSHXAxwYMeq3SLwAZn1Z5WmF1rIef1Cs5GjTapBg3IxNhLZ+Rulaqdo/lIo/MHoWeSUlCHtTxboNxINykR18Ab/PT/PM4LdfOZ0qPWxhnFXCrSmLHU/bHb83l3Z27K/b8zMI4tuZQCutXqOe+dKG1RSBNSjp6VK0IaZp3guCVEmbMrTpzOkUHWuSY8RA4I3hse7ZRgZlMWZ1t/1ftlPmpdeWo0+0rdf+6e10algL3+u3vZFbCgz1I3iuaICY1fVkqAHDMd0B2mFFtRGFfCwTgZSzB5RG5Y/eO1gtBj3Zf14oj8CqUq+Foo1AFXBPi42iPBST5imo6MHGbhaNbpF2L2D+0q0YKnBjAzRoKMUi7vAsrEvmYuZFOtnnC0tZgoD0V29s0sMwTejS7lUi0WiYgo3v68CnVOG1F9CPxV0d8oHf73NcMwM611rgZ2yLHQlQisCFHaZmtM+dS44BvawZK215tmMzVOCU48pWqB6NosdIksjxjth8wgmZ6guq0bOw7YUTjPBVLdUyNlzwRCf2OR1FYQZ02nz+Fa+YxM01gahkc7Uijxf+v7PPlE6xeXeQ+mSCOc2kyN/3dfCjCiR2suHhC+9yCzHYQ5tTEjMyyI01oA0xA1n/Dwm0xmxP4Ql+w6g0JNUdFY5TbRmA2Wfj7QkGUlpITHoRIZ2CJ6GO0yAhGbq0jMjAWd2YCzPz8CQgbb00O8kXlGRqn12GpfFGFwC0Iuf5sF9itgOev6P94ygfRJgvY9Sl5X4PXkFACpWNTds5fk/M6a7q2HgYPGZtu4nM6SaxtJJTQIlOKg+8wupNI4L49/PRIUbtUVGdKktcg5T0t9PNuNIOUvyYZgoy8BAibFEHV0U2cDzHjzdutLZfOqcLeDlH76MBYnzS+SzeMb+ww2Vh/jCkBshNXpdk4WYvdM93r3lzuYNXVkwmGGTiRGRlf60I2Dx+Yoren+0z6cg7pxReix0GqYSnpGX79g+tGdnfx1Ff6NM/Reak35VOpNWlmLik7QgBXndQ9K2A7E6RePQi0wUyBTLOwzZid+lZRCarlrTIwBrwDCjbHb2BFGbH5Y0QYUEcWOKe9wm2Tr+nUrfPzru33OXTAhJlfp4MGTY8t30eTqiE3Wh/G5RVwr34sH65/l9ZRgv8xs2PJlTTNDe2JGjk6NCzKA+0PcHRMSwgDXH0JdM79bikBF7ebqaesKBeE5QCs31WQbM09zuIk4fd6Wj0BeiB7Rn86TKDYW0EQm+cnDLVXOzLd1lGy2GLWPgQm2VWoLtntEV0yuundEab9e6i5SGky1WTZTm44PiD7Or4DT+9+KJFwsPmANp/ZhbiSNyWMQOMYMnvLf2suETBMNNEaMGI9irU4rqq6NDp2wInLbKGlQu8kpI9cxjunGF8cwYG+ruWP0d3yMC9bDhr49IYmcmKiVnYP7ac6+TqRlXNIx8J4mfOmsPFyKlI5OobPB2nM5KTRxwQHwnlQrrS55uKbv1/WG6K23dVyDv1fN44uv6+HkaAudjIvEgQiE6i5rawBCuZfMsHrCYc62xwKFbEPwU/hTpf2eyuVux1I8NrioGV9dZC4/nDr2xq/8pIwcO0SkG0HnUP7SULJD+I4Vb6YEmwnq+T/uBPWhxrrIuLmQebFQ6h0HCJuOrlBVUJRaRxn7wzVZeHnqbIcEePJcJSvi1giuIHINe3NJaVMmdzcuG8JsXcYHsJgMpPxmgjFqtukCQLzzOe22GOfqRyPsJGEbJ6zgUlQtQYPQ8+/SkzVldWrh8wuoNaX87gJPFnJuDKFQEC14DDy7eRVgrlblFYyzxsPcS7TtZp/PZJ/a9XYhz7V8XlFlZ8tj4SdwvkCLQFVfJPsFHBMBW6NkmkEwIx+3Ziag6gi8xZniBXGiBvBKROAQSFze3t8jFd+QJjYTmvD1TOEEG2XWLDG3ZBjYUO/FASQhUabTq/XBYl5F7nR5ReCQydECuUp1CIAtgPf4U+IDbhJ/mrxKQmaap2E310vHmaXj18bCMkagIqYZRJhnSPfwn/VdgfM/1aIzFxLv7ptPd/yMIE22mv19ACtAmpDf/ar4OOH5JCtBw2Df67k6Pv4t32LC8PCdB+A/jHNCW1whjGhIJybNigAU4+mOc/NgkuLu5KbqiXsEUm459Xvln1ez4OPImXxU+NEVIzOIG2JTte68+rMu7leQ++heWtXbFEiatTmFCjnsKYSUCNAtMg5zFrHcQbCvJrgtzm8S5aCZP2hDsP1kA4Y7bY3c1nTu/b1Xu6WpGOy1eSN0r0Bp003q6JUhnNYDkk3yafsTORodcIc2kM6sFAqyRhAZo5lIUfjrYL9ZCJKBC3sFiDxsxfHIMkvb0tnr1aeNGdqP0kg+TCky8CqNhqBkthO2Imn2F/S4VLMr6TMwSotNKR+9J2XZFuNRchVg+H0+tNRg8cWu3XFMYCxAXMPzmVa4V5j4B82jV2epMVpM/HFBZhSBXgMA9R2ttro1z4mXWFjIYWxmuOa3IAIHqe2YOavP4jDGrIlYvceoPhTxGmpyFTy+UPYGS5IEgu97OULnAbcr5LJLU54a4+7/yAxfHyqnfFYdmtsozv+kHUDaLANJmYDVAnkjpi85p7CSKt2f4UUfUzSz16DiEyuGBffHnEjcCtzo/cKM8r+6cb28WLYpjLzy7Yn5vsLo8MQ1Y86vhhsAZKCIMbA/zJkMFYWd9suntVZCyq9jMFsJJJQksZaAo9d+AW5U020vXf+z/Aw88XuTk7HGXAzGxn9xJo2Wjm6D3hoCItW0EqW1MgOfBbBkEj2Y9YYLb8tsXH1K3sK6Xn0bJIR/gZU6KlKSYOWnbgnIivjLSKO6p124iQu3Ru9wBQZvK2Uoq0umxnceDsoVvU+K57bWLntiIRIbRGb96UHoajTwteXl4gevW1aDzt7NcGDUBPPH5aGRmyG4VRpxQEsEk5j0/JCdRXcu4B0Cik/9huXuHkRnkHCZv54aKFjqXS2SPKG2WoXbTDEoBhndiFMnaTKV8fs2CejusKT5lLW/n4C6v3WzI2zQ4cr55qr8lkSu3WXvhwQFpZe59fdy//UBZ+bCZyXUzpNyeqMmBsYUPYeSxe01m31oyr2YgL+yOY2hZQGR/sTvzA70g1vfPveT783PMziwKX1wjrmkZnneiyOl6xcwyY26o3Czp5dpr8KEWbuBuJwVkfiEaUorujFeVVjsg7NjzUZOhp1z4KxQqBLCw98ma8DNPz2rWL733ldEXhRd/2EmtrYfZSa/iGUp5ZUu4O9FzD/RmSHy4h+IWFGP+NXHJ9fm0GxLg5IUC4gWLSU2TbFbmisWHsccd2xSGwg1iG9qtWTzJZrGwhD0tfvE66xDOhoss5fCiyJS5yBlPzQytI4Frsq+0upeTU/cXH2+aqkfIbCLd0Nesy/Cpx5/ifSkXaaTdQ7D617CVGBezQkrSHoIZcA0w+B9sSyLmIJhAHhxYMMsg+P/4eRI7xc1Ixg0Gwbs/ko4rqDVizbiCzdToMu4yYYweAD1qHTqj2c8dUqZqG/r82egTVDHYOQVruFif9xg4kyqb1szf0v+pDlm+fhwF4WLRwop7ypVETcUkLcPqtiswNEUtKhU5JRNI
Error 404 (Not Found)!!1
*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//[www.google.com/images/errors/robot.png](http://www.google.com/images/errors/robot.png)) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//[www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png](http://www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png)) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//[www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png](http://www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//[www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png](http://www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//[www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png](http://www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
404. That’s an error.
The requested URL /v1/projects/appthree-86b53/locations/us-central1/jobs/odds-ingest-orchestrator:run was not found on this server. That’s all we know.