I plan to create an aggregated sink from the whole organization and route it to 3 places at the same time:
Cloud Storage buckets - for compliance purposes
“Cloud Logging buckets” - to use “Logs Explorer”
In Splunk - to enhance information security
After reading the documentation and several attempts, I can only create 3 different sinks for each of these tasks and therefore I will have to pay for the same data 3 times even at the stage of collecting information at the “$0.50/GiB for logs ingested” tariff. Please tell me how can I reduce the costs so as not to pay 3 times? Is it possible to make one sink and configure 3 destinations in it?
Hello @SergeyN , I think that your cost assessment can be not entirely correct. Based on your description, I can assume that you define 3 sinks in each project from which you want to aggregate the logs. The total composition of the costs in this configuration will include:
Log ingestion to original project IF you store the logs in both locations: the original project and the aggregation project
Egress networking to Splunk endpoint
Log ingestion to aggregation project for logging bucket
Storage in the aggregation GCS bucket
The costs cover operations that you perform. If you decide to exclude the aggregated logs from being stored in the original projects then you can exclude the first bullet from the cost.
I am confirming would it work to have single sink for routing all logs toward aggregation project’s logging bucket and to have another 2 additional sinks defined in the aggregation project to export logs to GCS bucket and splunk. However, this change in the sink provisioning will not change the costs.
@leoy , I don’t understand. How can I set up one receiver (so as not to pay for receiving logs 3 times) and send it to 3 different places:> 1) gcs> 2) logging storage> 3) splunk> So far, I see only the possibility of making 3 different receivers for these purposes and therefore pay 3 times for receiving data.
I am not sure that I understand the term “receiver” that you use correctly. To define 3 different destinations to route your logs you have to define 3 different sinks. However, it does not mean that you will have to pay 3 times. Do you mean (by “pay 3 times for receiving data”) that you expect to pay 3 times ingesting costs if you define 3 sinks?
You can see in the Cloud Logging pricing summary that you are billed by logging data usage. In other words, routing feature is a complimentary service. You are not billed for using it no matter how many sinks you define (though the number is a subject to quota and limits). As I explained in my answer above, there is implicit use of other services that can cause billing such as GCS bucket storage, PubSub costs, egress costs, etc.
Perhaps I have a misunderstanding. When I read the documentation at the link https://cloud.google.com/stackdriver/pricing#logging-pricing-summary. I see the price for “Logging ingestion” ($0.50/GiB for logs ingested;). Please answer whether I will pay this money in each of the following cases:
Indeed the current documentation is a slightly ambiguous since it depends on the interpretation of what is considered “ingestion”. The “ingestion” means logs that are stored. Please note that defining a sink does not necessary means excluding the logs handled by the sink from being ingested. Read more about how to exclude logs from being ingested.
If you exclude the logs that are redirected by your GCS and PubSub sinks from being ingested into Cloud Logging storage, you will pay only GCS storage costs and PubSub usage costs which include throughput, storage (if you do not pull the message within 7 days) and egress costs.