APIGEE replacing systems like Datapower

As I understand APIGEE is only meant for API management with minimal functions for data transformation but I see XSLT policy, Java callout for other data transformation, Javascript policy for json transformation, SOAP to REST conversions in APIGEE. Can we replace systems like Datapower which is completely meant for transformation by APIGEE if performance can be compromised?

When there is lot of transformation in data format, its elements and attributes required, whether we can still recommend to replace Datapower with APIGEE?

3 Likes

there isn’t a simple answer - i think there will be lot more considerations than just the transformation functionality itself, for eg, two-speed IT - there was a recent webcast by @Dino and @Robert C. Broeckelmann Jr. regarding this - check out this webcast replay - http://apigee.com/about/resources/webcasts/modernize-soa-apis

1 Like

I don’t agree with your statement that “Apigee Edge has minimal functions for data transformation.”

I think Apigee Edge can transform data; you even cited a number of different possibilities. Now, the next question is, “Will the data transform capabilities in Apigee Edge be sufficient?” And to answer that, you need to get specific. What data transformations would you like to perform? XML to JSON? XSLT? JSON transforms? Something else?

**by the way, here’s a custom policy I recently built for one limited case of XML transforms.

2 Likes

I think it is safe to try. You can run both easily enough. Should be possible to compare. I would look at more than just performance; consider ease of change, etc.

LOL… I think Apigee Edge can transform data too.

1 Like

Please define ‘a lot’ of data transformations. Maybe if you provided a few more specific examples I’d could let you know if it’s feasible. I have some exposure to both solutions but less so on Datapower.

1 Like

@Dino so now i have to ask - since this is a java callout - does that mean it doesnt run as a shared resource (using full memory, etc every time its called) ?

Hi @Benjamin Goldman - I don’t quite understand the question you’re asking. This thing is a Java class that runs as a custom policy in the Edge message processor. Edge loads it via a separate classloader, but the class runs in the Java VM, alongside all the other classes for other policies, including the builtin policies. I don’t know what you mean by “using full memory, etc”. Sorry to be dense! But if you elaborate I’ll try to answer.

ref: Apigee Edge docs.

We’ve almost complete with our migration from DP to Apigee Edge and have been pretty happy so far mostly from a sheer visibility perspective. We didn’t do any significant level of transformations in DP and it was mostly a facade to Tibco. Like the previous commentors have mentioned, Apigee definitely supports transformations but it’s not clear what specifically you’re looking to do. I’ll say one of the best things I like about being on Apigee from DP is the analytics and visibility into what traffic is passing through the platform. With DP we could see there was traffic but figuring source/destination or even which operations were being used was a complete nightmare. We hired consultants just to help us figure out that part alone…imagine hiring an Apigee consultant to figure out the traffic patterns of your proxies.(ridiculous)

So to answer your question directly, yes, you can replace DP with Apigee and you’re on the right track if you’re considering that path.

3 Likes

Great feedback @Alan Williams , Glad to know analytics is helpful.

Thanks for your comments.

Huge data transformation from SOAP XML to JSON.Many elements names and their attributes name also have to be changed.

I thought this might be the case. Many of the use cases we use Apigee for involve converting large complex SOAP web services to Hypermedia inspired REST facades.

In terms of HUGE… can you give me an approximate size of the XML payload in bytes and number of elements? Also what sort of TPS would you be expecting during peak traffic?

I don’t have all the details so I feel bad asking this question, but why? If you’re only converting from massive SOAP/XML to massive JSON, I’m not sure I get the point. Should be easy enough to do with Edge or DP, and if performance is acceptable you could probably eliminate DP from doing that, for reasons others have suggested here around ease of use, maintenance, etc.

On the other hand, if you’re taking a huge chunk of SOAP/XML and thinking about turning it into multiple, more useful JSON web apis, then full steam ahead and Edge is great at doing that. :wink:

I think you could use XSLT for that purpose - transforming and changing names. If you don’t like that approach, and prefer to code in JS rather than XSLT, then you could use the out-of-the-box XMLToJSON policy and then write a JS policy that uses Array.map and Array.filter. I find it much easier to do this way. It depends on the particulars of your case though - the size of the payload, the request rate, the familiarity and comfort you have with XSLT, and so on. Maybe you even already HAVE the XSLT written? (If you’re a Datapower shop, then this is not far fetched). In which case, just use it within Edge.

But anyway, what you’re describing sounds pretty mainstream.

Carlos, I didn’t ask the question and I don’t have all the details, but some people want to convert from a Massive SOAP/XML with all the soap envelope stuff, to a JSON, and at the same time, insert default values or “values from partner profile” into the JSON.

What I mean is, the app sends in a slim JSON, the Edge policies convert to XML, then augment that payload with defaulted values from the partner profile (looked up from API key or product ID, etc) and then wrap that all in the correct SOAP envelope.

Even this makes the job of the consuming developer much simpler.

But I agree with your idea that there are huge opportunities for getting more agile, if people can just imagine going beyond the basic “SOAP-to-REST” conversion concept.

Hello RK4. Sorry I couldn’t respond yesterday. As noted above, Dino and I touched on the topic of datapower vs. Apigee Edge in our webcast last month.

Performance does tend to be important, but as several other comments point out there are potentially numerous other factors. Most shops that use datapower likely have a sizeable investment in it. Does it make sense to the business to do a wholesale replacement from an ROI perspective? The answer will vary depending on the organization’s circumstances.

Performance for data transformations on DataPower vs. Apigee Edge depends on a variety of factors. So, it is hard to make any general observations. For pure XSLT transformations on a datapower hardware appliance, there is likely to be a performance and scalability benefit–that is what the thing was invented for. The moment you move that idea onto DataPower virtual edition, the performance aspects will likely see a more level playing field. But, all of this depends on the size of the XML document being manipulated, complexity of the XML structure, how much of it is being manipulated, how much other security processing is also occurring, etc. For JSON transformations or JSON<->XML transformations, results will definitely vary depending upon the complexity of the data structures and other factors mentioned. So, bottom line, one has to try it with their specific use cases.

Likewise, on the Apigee Edge side, on-premise vs. cloud will likely have different performance characteristics depending upon the on-premise hardware platform that is running the VMs. On Apigee Edge public cloud, the public/free/trial version is probably going to behave differently in most situations than the dedicated/paid/enterprise version. Again, needs to be tested with realistic data sets.

It also depends on what patterns you are using datapower for. Is it your internal ESB? Is it acting as a gateway (security and interface wise) to the internal ESB? Or, gateway in the DMZ? Is it dealing with data transformations whose inputs/outputs are something other than JSON or XML/SOAP? If so, that may not necessarily be the use case that Apigee Edge is targeted towards–Apigee Edge can run custom Java code, so just about anything is possible, but again, not necessarily a targeted use case. Apigee Edge could act as a replacement or alternative gateway to an existing ESB on the internal network. It is also very common to see it as the gateway in a DMZ. It depends who the actors are that are invoking these APIs.

Summarizing, at a very high-level, I’ve been in datapower shops where I honestly believe Apigee was the better strategic fit and replacing it makes sense. I have also been in shops where keeping datapower to let it do the things it is good at or run a lot of legacy stuff that it made no sense to migrate in the short-to-medium term was the wise choice–but, augment it with the things that Apigee is good at. The webcast describes such a situation–obviously, there are costs associated with this that must be weighed. I could also envision scenarios where datapower is probably the better choice–but, this is an apigee forum.

There is also the API Management angle(governance, life-cycle management, development community, etc), are you using datapower as part of an API Management solution already? Or, it’s more traditional use cases? Is Apigee Edge in your organization being used for the full API Management story or is it more about being a datapower replacement? Many ways this can go…ties back to your use cases/requirements and what problem(s) you are trying to solve.

If you have more specific details about your scenario, I’ll be happy to respond back.

RCBJ

2 Likes

Thanks @Robert C. Broeckelmann Jr. for your perspective! This forum is intended to provide broad views around APIs and digital, including API management. It’s totally fine to have discussion beyond Apigee :slight_smile:

Hi @Robert C. Broeckelmann Jr.

thanks for providing such a detailed reply.

In our scenario,Datapower is used mainly for transformations.And consumers send SOAP messages to consume the services through datapower.

Now,the consumers want to consume services as REST calls and APIGEE comes into picture there as they also want API management layer.

APIGEE not only has to do mere xml(SOAP) to json transformation but also has to pick and restructure only necessary fields from xml before converting to json.

So,I think XSLT will be better choice to convert SOAP message to another xml.

But some XMLs are big and complex.So,XSLT is not good performance wise.Is there any other way I can do this transformation?Or XSLT is better choice?

But some XMLs are big and complex.So,XSLT is not good performance wise.

Don’t be so sure about that! Before making that conclusion, i suggest you test it.

Sure @Dino .In Message broker tool,XSLT is not preferrable as it slows down the performance.I thought the same here.