MessageLogging Policy - Logstash

Google Cloud Apigee
1

Hello,

Right the issue is that I’m not seeing messages from the MessageLogging policy on a basic ELK stack that is running on a remote machine. (Using Apigee Edge Cloud FYI)

I’ve set up a new API Proxy, which is essentially a ‘Hello World’ nodejs example which I’ve then attached a MessageLogging policy to it’s Proxy Endpoint, Post Flow Response.

As follows:

    <PostFlow name="PostFlow">
        <Request></Request>
        <Response>
            <Step>
                <Name>Message-Logging-LogStash</Name>
            </Step>
        </Response>
    </PostFlow>

And the Policy is set up as:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="false" enabled="true" name="Message-Logging-LogStash">
    <DisplayName>Message Logging-LogStash</DisplayName>
    <Syslog>
        <Message>Message.id = {request.header.id}</Message>
        <Host>aremotehost</Host>
        <Port>443</Port>
        <Protocol>TCP</Protocol>
    </Syslog>
</MessageLogging>

(Note I’ve masked the remote host and the ootb default message)

On the listening remote host a proxy is set up within the webserver that is listening and sends the call to the ELK stack; this has been tested and works as expected; I can make calls to it and see the message’s displayed via Kibana. This has been tested internally and externally.

When I trace the sending a request it seems to report that everything is fine and that the MessageLogging policy is getting a 200 response

1331-screenshot.png
1331-screenshot.png1140×220 8.58 KB

Note: that I have also modified the host to something that does not resolve and had no luck, in fact Apigee reports 200 through out.

Looking around, I’ve seen that there have been a few successful attempts at this, such as here

Now I’ve tried the following Logstash input types:

  • syslog
  • tcp
  • http

All work as expected when making calls NOT via Apigee; but I’m unable to see anything from Apigee.

  1. Is there anything obvious I’m missing?
  2. Is there a way to see a more verbose trace/output/logging when using Apigee Edge Cloud?
  3. How can I be certain that the MessageLogging policy is sending message as expected?

Thanks

Alex

2

Hi @alex.tylor , thanks for the comprehensive report of this issue. Although I haven’t tried the integration of MessageLogging Policy with Logstash and because you mentioned that you’re leveraging Node.js, I’d encourage you to also try Winston-Logstash from Node.js. The configuration of Winston is simple and I already tested it with Winston Loggly, which essentialy would be a different connector in Winstor. You can find an article in the community explaining how to get started with it here: https://community.apigee.com/articles/3352/api-log-management-push-model.html

Please let me know if you have any questions.

@Arghya - any leads on MessageLogging issue?

3

Hi @Diego Zuluaga

The use of nodejs was simply to create an API proxy quickly with a response; in order to test the MessageLogging Policy with Logstash. (Although I had come across the above article, and does look interesting, and likely test that out at a later date.)

4

What he is trying to do is actually rather smart: apigee syslog log policy => https proxy => logstash syslog listener. This is one combination we decided NOT to test beacause we really like having super safe async options in the middle of message flow.

A debug step: can you try bypassing the proxy and just send a couple of plain text messages to the logstash syslog listener? That way you can sit on logstash in debug and watch the messages come in start zeroing in on what might be wrong?

5

this has me interested enough that i might try to set this up tonight and see if I can make it work.

6

I can by pass the https proxy, and see messages come through (not externally due to network restrictions), and externally via the https proxy. I’m fairly sure that there is an issue sending the messages from apigee to my listening web server… but don’t know how to prove it within Apigee :S I can be certain that nothing is being reported in either the access or error log of my web server instance.

7

This is what I expected. are you using a self signed cert or something on the proxy? Is it possible to do this test w/ a valid wildcard cert or something? (hostname valid) Just to rule out BS inside the https stack on apigee?

8

@alex.tylor

Just a couple of questions:

  • Are you sure that 443 is the correct port that your logstash server is listening on, can you change that to something else and try ?

  • Do you have any firewall settings on the server hosting logstash, which may be preventing Apigee instances to talk to the logstash server ?

You would not get this information by simply looking at the trace, we would have to dig into the system.log on the message-processors. You can also email me your org-name, proxy and other details so that I can take a look, or open a support ticket with Apigee whereby they can provide more details and log entries to troubleshoot the issue.

9

If you are using 443, I understand you are using SSL. If it is true, you may want to change the policy to have tag as below:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="false" enabled="true" name="Message-Logging-LogStash">
    <DisplayName>Message Logging-LogStash</DisplayName>
    <Syslog>
        <Message>Message.id = {request.header.id}</Message>
        <Host>aremotehost</Host>
        <Port>443</Port>
        <Protocol>TCP</Protocol>
	<SSLInfo>
              <Enabled>true</Enabled>
        </SSLInfo>
    </Syslog>
</MessageLogging>
10

This looks really promising!

11

Right sorry for the radio silence… firstly thank you all for your help. My PoC works; so let me out line what’s going on.

(Rather crude setup outline)

   Int           Firewall       Ext         
                    +                       
+---------------+   |                       
|               |   |                       
| Local Machine |   |                       
|               |   |     +----------------+
| +-----------+ |   |     |                |
| |           | |   |     | Apigee Message |
| |  haproxy  +-----------+ Logging Policy |
| |           | |   |     |                |
| +----+------+ |   |     +----------------+
|      |        |   |                       
| +----+------+ |   |                       
| |           | |   |                       
| | ELK Stack | |   |                       
| |           | |   |                       
| +-----------+ |   |                       
|               |   |                       
+---------------+   +

The ELK stack is essentially OOTB, the only thing to note is the Logstash instance config:

input {
    syslog    {
        type=>"syslog"
        port=>5544
        }
    }
output {
      elasticsearch { host => localhost }
        stdout { codec => rubydebug }
}

The Apigee Message Logging Policy is as follows:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="false" enabled="true" name="Message-Logging-LogStash">
    <DisplayName>Message Logging-LogStash</DisplayName>
    <Syslog>
        <Message>message = {request.header.id:unknown_apigee_request_header_id}</Message>
        <Host>apigee-logger.domain.com</Host>
        <Port>443</Port>
        <Protocol>TCP</Protocol>
        <SSLInfo>
            <Enabled>true</Enabled>
        </SSLInfo>
    </Syslog>
</MessageLogging>

HAProxy was installed, and certs created (as this was just a PoC dummy certs were made and used)

listen logstash
   bind *:443 ssl crt /etc/ssl/domain.com/domain.com.pem
 
   mode   tcp
   option tcplog
 
   server log1 127.0.0.1:5544

Note that ‘apigee-logger.domain.com’ 443 traffic was routed to the local machine on 443.

Then using the ‘trace’ function we can make the calls and see:

haproxy logging:

Oct 14 16:25:57 localhost haproxy[7187]: 54.208.24.86:52526 [14/Oct/2015:16:25:52.881] logstash~ logstash/log1 199/1/4276 0 -- 0/0/0/0/0 0/0

Logstash Logging:

{
           "message" => "<14>Wed Oct 14 15:37:43 UTC 2015Info: message = unknown_apigee_request_header_id\u0000\n",
          "@version" => "1",
        "@timestamp" => "2015-10-14T15:37:44.002Z",
              "type" => "syslog",
              "host" => "127.0.0.1",
              "tags" => [
        [0] "_grokparsefailure_sysloginput"
    ],
          "priority" => 0,
          "severity" => 0,
          "facility" => 0,
    "facility_label" => "kernel",
    "severity_label" => "Emergency"
}

So that’s kinda good I guess :slight_smile: