Dual Data Center Analytics Group Setup

anonymous · March 25, 2016, 7:06pm

Hi everyone,

I have just set up a Dual data center Apigee Edge on-prem deployment, wherein our preexisting DC was expanded to encompass another. We created another environment, prod-2, so that there could be targetserver and virutalhost config independence via the add-env.sh script.

When I ran the enable-ax.sh script for this new env, I saw this message at the end:

--> curl -X POST [http://mgmt-svr:8080/v1/organizations/org/environments/prod-2/analytics/admin?type=organizations](http://mgmt-svr:8080/v1/organizations/org/environments/prod-2/analytics/admin?type=organizations) [...]
{
  "code" : "PG e670d246-bf7e-4217-bf26-15cd8ddb8eff Not registered with any group",
  "message" : "PG e670d246-bf7e-4217-bf26-15cd8ddb8eff Not registered with any group",
  "contexts" : [ ]
}
==> 400

Whenever I try to look any anything analytics related in the management ui, I see the red error box with “Could not resolve scope ..”

We also see the following error in the PG system logs:

2016-03-25 14:15:18,058  pool-30-thread-1 WARN  c.a.a.m.MessageConsumerBuilder - MessageConsumerBuilder.isQpidInSameRegionAsSelf() : Not opening consumers to 71be1378-b2ac-4e1d-b68f-f60afcf2ed6e, from this server 39213ee3-dbfa-4c49-bdc8-26851e1e542b as they are in different regions dc-1/dc-2

All that being said, I have a more general question: If we want to have separate analytics in the mgmt ui for our prod and prod-2 environments, should we create a different analytics group? What are the best next steps here for enabling analytics for our new prod-2 env which are separate in the ui from the prod env analytics?

Thanks a lot!

Best,

Chris

anonymous · March 26, 2016, 8:09pm

Hi Chris,

Please run the following command to validate the analytics groups scopes:

curl -v -u <user>:<password> "http://<management-server-ip>:8080/v1/analytics/groups/ax"

The curl, will return something like:

[ {

“name” : “axgroup001”,

“properties” : {

“consumer-type” : “ax”

},

“scopes” : [ “abc~dev”, “abc~qa” ],

“uuids” : {

“aries-datastore” : ,

“postgres-server” : [ “e8ca04bb-ec81-4ca4-bd5c-cf8d17f141ec:a576159f-33fc-41ee-9e53-969beab9bef1” ],

“dw-server” : ,

“qpid-server” : [ “f44d25b4-c644-4239-871d-d83674824e4e”, “8c158a3d-d57c-4e97-b8da-9510b7ffc911” ]

},

“consumer-groups” : [ {

“name” : “consumer-group-001”,

“consumers” : [ “f44d25b4-c644-4239-871d-d83674824e4e”, “8c158a3d-d57c-4e97-b8da-9510b7ffc911” ],

“datastores” : [ “e8ca04bb-ec81-4ca4-bd5c-cf8d17f141ec:a576159f-33fc-41ee-9e53-969beab9bef1” ],

“properties” : {

}

} ],

“data-processors” : {

}

Question:

Are MPs on the new DC also associated to the old Environment?, is that what you are looking to do? or will MPs on the new MPs only be associated to the new environment?. What do you specify as part of add-env.sh?

anonymous · March 28, 2016, 4:42pm

Hi Maudrit,

Thanks for the tips. I’m waiting to get access to the management server port 8080, but I can answer your questions now:

We want the DC1 MPs to be associated with only the old environment (prod), and the DC2 MPs to be associated with only the new environment (prod-2). In the add-env.sh, I made sure to only add the 2 MPs from DC2 to the new environment (prod-2).

I’ll post the output from the curl shortly. Thanks again for your help!

Best,

Chris

anonymous · March 28, 2016, 5:32pm

Here is the output to the curl. I notice that there is no mention of our new environment, prod-2:

[
  {
    "name": "axgroup001",
    "properties": {
      "consumer-type": "ax"
    },
    "scopes": [
      "Intralinks~prod"
    ],
    "uuids": {
      "aries-datastore": [],
      "postgres-server": [
        "c5590478-e7d6-47d4-b921-a50ea1daf229:b28b0f8e-5a5b-4b40-bb54-4de5f0bfc687"
      ],
      "dw-server": [],
      "qpid-server": [
        "71be1378-b2ac-4e1d-b68f-f60afcf2ed6e",
        "dfabb5c3-9b71-4b95-a64d-ebfafa4b181d",
        "7c516df1-1f1d-4487-852c-fe9c6cbd8628",
        "39213ee3-dbfa-4c49-bdc8-26851e1e542b"
      ]
    },
    "consumer-groups": [
      {
        "name": "consumer-group-001",
        "consumers": [
          "71be1378-b2ac-4e1d-b68f-f60afcf2ed6e",
          "dfabb5c3-9b71-4b95-a64d-ebfafa4b181d",
          "7c516df1-1f1d-4487-852c-fe9c6cbd8628",
          "39213ee3-dbfa-4c49-bdc8-26851e1e542b"
        ],
        "datastores": [
          "c5590478-e7d6-47d4-b921-a50ea1daf229:b28b0f8e-5a5b-4b40-bb54-4de5f0bfc687"
        ],
        "properties": {}
      }
    ],
    "data-processors": {}
  }
]

anonymous · March 30, 2016, 2:53pm

Hi @Maudrit,

Any ideas as to what we can do to enable analyics? Thanks!

Best,

Chris

paulmibus · March 31, 2016, 1:56pm

From the output of the AX group, it looks like e670d246-bf7e-4217-bf26-15cd8ddb8eff is an extra PGSQL UUID floating around. Could be from an old component that was removed or from an installation restart. You can delete that UUID (if not needed) by calling:

> curl -iu <creds> [http://<msip>:8080/v1/servers/e670d246-bf7e-4217-bf26-15cd8ddb8eff](http://<msip>:8080/v1/servers/e670d246-bf7e-4217-bf26-15cd8ddb8eff) -X DELETE

anonymous · March 31, 2016, 8:47pm

Hi @Paul Mibus

Thanks for the help. Is there any way to conclusively ascertain whether this UUID is not needed? I want to make sure of this before deleting it from the servers inventory. Thanks!

Best,

Chris

paulmibus · March 31, 2016, 8:50pm

That will vary a bit by component. In this case since we’re dealing with a Postgres Server, if the UUID does not appear in any analytics groups you are safe to remove it. To check all AX groups, do a GET on /v1/analytics/groups/ax. If you don’t see the UUID anywhere in the output then that server is not in use.

You can also check each service in your cluster for their self-reported UUID: GET localhost:/v1/servers/self/uuid. The ports are 8080 for management server, 8081 for router, 8082 for MP, 8083 for Qpid Server and 8084 for Postgres Server.

anonymous · April 1, 2016, 9:13pm

Hi Paul,

After removing the UUID from the registry and rerunning the enable-ax.sh script, I get the same error but with a different UUID (the other of 2 PG servers):

--> curl -X POST [http://localhost:8080/v1/organizations/foobar/environments/prod-2/analytics/admin?type=organizations](http://localhost:8080/v1/organizations/foobar/environments/prod-2/analytics/admin?type=organizations) [...]
{
  "code" : "PG 2c15a8fd-e8c2-40be-863a-fbc9c454b436 Not registered with any group",
  "message" : "PG 2c15a8fd-e8c2-40be-863a-fbc9c454b436 Not registered with any group",
  "contexts" : [ ]
} 
==> 400

Looks like something else is the matter with the enabling of ax. Any other possible issues come to mind? I do see that our prod-2 environment is not in the scope of axgroup001, which is our only ax group. The way I see it, we can do one of two things:

Create a new ax group for prod-2, add all the necessary ax servers to the group, and add the environment prod-2 to its scope.
Add the two new PG servers (and possibly two new Qpid servers?) to the existing axgroup, and then add prod-2 to its scope.

Let me know if you have any thoughts/concerns/ideas. Thanks!

Best,

Chris

anonymous · April 4, 2016, 4:33pm

this is fixed. the steps were as follows:

run the following two curl commands (in order):

curl -v -u <creds> -X POST -H 'Accept:application/json' -H 'Content-Type: application/json' "http://<msip>:8080/v1/analytics/groups/ax/axgroup001/servers?uuid=<pg_master_uuid>,<pg_slave_uuid>&type=<serverType>"

curl -v -u <creds> -X POST -H 'Accept:application/json' -H 'Content-Type: application/json' "http://<msip>:8080/v1/analytics/groups/ax/axgroup001/consumer-groups/consumer-group-001/datastores?uuid=<pg_master_uuid>,<pg_slave_uuid>"

run the enable-ax.sh script on the management server: sh /opt/apigee4/bin/enable-ax.sh

paulmibus · April 4, 2016, 5:03pm

One more thing that you may have to do if it was not already done is to enable master/slave replication at the PostgreSQL database level. The analytics group update would make the Apigee Edge platform aware of an existing master/slave pair, but it will not actually enable the database replication. You can find the steps to do that on page 64 of the Apigee Edge Install and Configuration Guide.

anonymous · April 4, 2016, 5:20pm

Hi @Paul Mibus,

Thanks for the tip! I enabled m-s replication during the initial install. What’s strange now is that even though the scope error is gone in the mgmt ui, there does not appear to be any data coming through. The charts in the dashboards are all saying “no traffic in selected range” even though I have been invoking steady API traffic all day.

Is there a reason why the UI would think that there is no data when in fact there is?

Thanks!

Best,

Chris

anonymous · April 4, 2016, 8:49pm

Hi @Paul Mibus,

I’ve just noticed that the “could not resolve scope” error message still persists but only in the DC-1 Management Servers. The steps outlined in my previous comment only resolved the error message on DC-2 Management Servers.

What’s more is that I do not see the updates in the analytics group when I hit the DC-1 management server api. I effectively see different configurations depending on whether I hit the management api on DC-1 or DC-2. How could this be?

paulmibus · April 4, 2016, 10:46pm

Unfortunately if you are seeing different data in each region you may have two disconnected regions rather than a single unified planet. It is possible it could be something simpler, but I recommend an audit of your installation from the ground up, starting with validating that your installation response files, if any, correctly reference peer data centers as described in the Apigee Edge for Private Cloud Install and Configuration Guide.

Topic		Replies	Views
On-premises Edge 4.16.01 - Analytics for new Environment. Apigee private-cloud-deployment	11	17	March 16, 2017
PG <> found in more than one group Apigee api-runtime	9	5	March 28, 2020
Working with the Analytics Group API Apigee	4	11	July 21, 2016

Dual Data Center Analytics Group Setup

AI Suggested topics