How to Monitor apigee logs with Elasticsearch and logstash

Google Cloud Apigee
1

Are you using Apigee in the cloud or on-prem ? What kind of logs are you asking about ? i.e. system logs or audit logs ?

2

you could write logs form Apigee to a syslog destination using the MessageLogging policy

http://apigee.com/docs/api-services/reference/message-logging-policy

So logstash already supports syslog as input,

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html

so it should be fairly straightforward to configure syslog input in logstash and write logs from Apigee

Thanks,

3

Hi Mukundha,

I am basically new to apigee and kinda learning how the MessageLogging policy works. I have created a simple yahoo weather API Proxy and trying to forward the logs to our syslog server. The syslog server i created is working fine as i have a rsyslog client that can send data to it and i can view them in kibana. (ELK setup).

Below is the message policy i am using.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?> rsyslog Message.id = {request.header.id} xx.xx.xx.xx 5544 TCP

But i don’t seem to find any logs been received and seen in Kibana.

4

isn’t syslog UDP?

plus, also check if firewall rules has been set to open port 5544 [either tcp or udp, based on your config]

5

It supports both UDP and TCP. And the example in Apigee Message policy documentation also describes it.

6

not sure if you missed this -“plus, also check if firewall rules has been set to open port 5544 [either tcp or udp, based on your config]”

also, are you trying this in the cloud? can you share which org it is?

7

Yes i have opened the 5544 port to allow traffic from the Public Message Processor which we have got.

This is Apigee Edge , and the ELK setup is on AWS.

The org is hpcorp.

8

can you share your proxy? could be a proxy config issue - lets try to rule that out

9

What do you want me to share in that. ? It is a yahoo weather API.

Is this what you require.

https://hpcorp-dev.apigee.net/weather/forecastrss?w=2502265

10

@mukundha@apigee.com I can receive the syslog messages now. Can you guide me on how do we record the API status codes. Like 200 OK or a 404 error in the messages. Something like APIResponseCode.

11

@Niraj - the variable that you need to use is response.status.code. So in message logging policy, Message tag, include this in curly brackets (as you are in interested in the value of the variable):

<MessageLogging name="LogToSyslog">>
  <Syslog>
    <Message>{response.status.code}...</Message>

Here is the full documentation of message logging policy: http://apigee.com/docs/api-services/reference/message-logging-policy

Obviously in order for this to work, you will need to put this policy in response flow - otherwise response variables will not be available for you.

All other variables that you might want to use is explained here: http://apigee.com/docs/api-services/reference/variables-reference. Just reference them in curly brackets and you are good to go.

2 Likes
12

can you also post what was the issue you were facing earlier?

13

@mukundha@apigee.com The problem was something weird. I had the same rsyslog policy been applied to request and response and when i removed one of them. It started working. Don’t know but from the ELK side i didn’t had anything been changed.

Can we apply the same policy to both request and response?

14

@Ozan Seymen Thanks for the wonderful answer. I really helped me a lot to gather all the variables information at a single page. But i was unable to collect the response code as mentioned by you. I am using a single policy for both request and response. Is that something supported or we need to configure two different policy for the same.

15

Hi @Niraj - can you elaborate on what you mean by “unable to collect the response code” please? Are you getting an error from Apigee or you are unable to see correct data?

It would also help if you can paste your message logging policy and the resulting data you are seeing on your syslog servers.

16

@Ozan Seymen I don’t see the status code in my kibana dashboard. There is no data been populated and the message field is blank.

17

Hi Niraj, I guess your policy is getting executed on the request flow, you just need one policy attached at the Response flow to send all the information to syslog. If you see issues, check the Trace in apigee as well, to make sure if those variables are populated.

18

Niraj, can you please provide more info when you post the question. This will help people understand your thought process and provide you a good answers much faster.

19

Hi Mukunda, Today i tried adding a new message logging policy and attached that to the response flow as well. Saved it and tried firing the URL , but i see no response code in my kibana. Am i doing something wrong here. Apologies for my basic questions…

20

Hi Sriki77, It was more of question rather than problem. So i had no more info to post. Let me know if you need more info on this.