Hi,
I am currently experimenting with the liveAgentHandoff Feature and Voximplant and noticed that when the liveAgentHandoff gets triggered, my inputs still get recognized by Voximplant but Dialogflow gets stuck and wont react or continue the flow. Also sending an Event to Dialogflow doesn’t have any impact. How can I resolve that issue, does Dialogflow just stop by default when the liveAgentHandoff gets triggered or is the Voximplant connection the problem?
Thank you!
// You can try this code
conversationParticipant.addEventListener(CCAI.Events.Participant.MarkerReached, (e) => {
Logger.write("MARKER REACHED IN USER VE SCRIPT")
// some flag that set for live agent handoff
if (transfer) { call.stopMediaTo(conversationParticipant);
// Send the Event to Dialogflow after media has stopped flowing
conversationParticipant.analyzeContent({ eventInput: { name: "WELCOME", languageCode: languageCode } });
}
})
Thanks, but I already use this code. It seems, the Dialogflow Event handler does not recognize the eventInput of Voximplant, because I can see in the log that the event gets executed and als the WELCOME event works to kick of the conversation.
It will recognize it if you use this exact code, not just analyzeContent function. You need to stop media to DF in MarkerReached event, since when the media is flowing via GRPC DF doesn’t recognize commands.
conversationParticipant.addEventListener(CCAI.Events.Participant.PlaybackFinished, (e) => {
if (transfer) {
transfer = false;
/* callPSTN calls the first phone number and gets the phone number dynamic from the incoming call from call.callerid()
The first phone number has to be verified by Voximplant/Settings/CallerID */
outboundCall = VoxEngine.callPSTN('+exampleNumber', call.callerid() );
Logger.write("executed calllPSTN")
outboundCall.addEventListener(CallEvents.Connected, () => {
Logger.write("call connected")
call.stopMediaTo(conversationParticipant);
conversationParticipant.analyzeContent({
eventInput: { name: "TRANSFER_SUCCESS", languageCode: languageCode },
});
endConversation();
});
outboundCall.addEventListener(CallEvents.Failed, (e) => {
Logger.write(`Transfer failed: ${JSON.stringify(e)}`)
call.stopMediaTo(conversationParticipant);
conversationParticipant.analyzeContent({
eventInput: { name: "TRANSFER_FAIL", languageCode: languageCode },
});
call.sendMediaTo(conversationParticipant);
});
}
else if (hangup) {
endConversation();
}
});
Tried it like this now, but it does not work, the event gets sent according to the log but the call remains silent but connected and the dialogflow fulfillment dialogue is not triggered.
Also I’ve checked the names of the Event sent and the Events in DF are the same.
Any other idea where the problem may is?
Hi, the problem is that you have
call.stopMediaTo(conversationParticipant);
in the wrong event, this is being overriden in
CCAI.Events.Participant.MarkerReached
So you need to add this handle and call stopMediaTo there
Did you get it working? I have exactly the same problem. I’ve listened for the MarkerReached event, and called stopMediaTo, but it makes no difference.
conversationParticipant.addEventListener(CCAI.Events.Participant.MarkerReached, (e) => {
Logger.write("MARKER REACHED IN USER VE SCRIPT")
if (transfer) {
call.stopMediaTo(conversationParticipant);
}
});
conversationParticipant.addEventListener(CCAI.Events.Participant.PlaybackFinished, (e) => {
if (transfer) {
Logger.write("Do transfer");
transfer = false;
// Do an outbound call and connect it with the inbound one
outboundCall = VoxEngine.callSIP(SIP_CONNECTION, {
authUser: USERNAME,
password: "wrong_password",
extraHeaders: {},
video: false,
outProxy: null
});
outboundCall.addEventListener(CallEvents.Connected, () => {
Logger.write('connected');
VoxEngine.easyProcess(call, outboundCall, () => {
conversationParticipant.analyzeContent({
eventInput: { name: "TRANSFER_SUCCESS", languageCode: languageCode },
});
endConversation();
recorder.stop();
});
});
outboundCall.addEventListener(CallEvents.Failed, (e) => {
Logger.write(`Transfer failed: ${JSON.stringify(e)}`)
conversationParticipant.analyzeContent({
eventInput: { name: "TRANSFER_FAIL", languageCode: languageCode },
});
});
}else
if (hangup) {
endConversation();
}
})
You can use Logger.write and check in the log when stopMediaTo and startMediaTo are being called during the scenario execution. When media is going to the DF agent it doesn’t detect events, although GRPC allows this, but DF doesn’t support this. In your scenario it’s likely that the order of calls to start/stop media function is wrong
Hi,
I tried to put the stopmedia in different places but the logging looks right to me:
this is the part before the sip call:
2023-03-21 15:26:39 Agent handoff
2023-03-21 15:26:39 Executing JS command: StopMedia with params [{from = A111E0CD26143ED7.1679412381.1949363 ; to = L6rtn7f8QUqRtzHQuWVy2i4OUdglnkaRq7sollbMVew ; } ; ]
2023-03-21 15:26:39 Sent event to JS onPhoneEvent with params [{id = Le3FtBGxRIeeijnq6W0EVYg5dRYGREIql9abd_2WRbM ; name = AI.Events.CcaiParticipantPlaybackStarted ; participantId = L6rtn7f8QUqRtzHQuWVy2i4OUdglnkaRq7sollbMVew ; playbackId = d9b39eb8-3b77-4eae-a8e7-207bde2ca262 ; } ; ]
2023-03-21 15:26:40 Sent event to JS onPhoneEvent with params [{id = Le3FtBGxRIeeijnq6W0EVYg5dRYGREIql9abd_2WRbM ; name = AI.Events.CcaiParticipantMarkerReached ; offset = -500 ; participantId = L6rtn7f8QUqRtzHQuWVy2i4OUdglnkaRq7sollbMVew ; playbackId = d9b39eb8-3b77-4eae-a8e7-207bde2ca262 ; } ; ]
2023-03-21 15:26:40 Executing JS command: SendMedia with params [{from = A111E0CD26143ED7.1679412381.1949363 ; to = L6rtn7f8QUqRtzHQuWVy2i4OUdglnkaRq7sollbMVew ; } ; ]
2023-03-21 15:26:40 MARKER REACHED IN USER VE SCRIPT
2023-03-21 15:26:40 Executing JS command: StopMedia with params [{from = A111E0CD26143ED7.1679412381.1949363 ; to = L6rtn7f8QUqRtzHQuWVy2i4OUdglnkaRq7sollbMVew ; } ; ]
2023-03-21 15:26:41 Sent event to JS onPhoneEvent with params [{id = Le3FtBGxRIeeijnq6W0EVYg5dRYGREIql9abd_2WRbM ; name = AI.Events.CcaiParticipantPlaybackFinished ; participantId = L6rtn7f8QUqRtzHQuWVy2i4OUdglnkaRq7sollbMVew ; playbackId = d9b39eb8-3b77-4eae-a8e7-207bde2ca262 ; } ; ]
2023-03-21 15:26:41 Sent event to JS onPhoneEvent with params [{id = Le3FtBGxRIeeijnq6W0EVYg5dRYGREIql9abd_2WRbM ; name = AI.Events.CcaiParticipantPlaybackStopped ; participantId = L6rtn7f8QUqRtzHQuWVy2i4OUdglnkaRq7sollbMVew ; playbackId = d9b39eb8-3b77-4eae-a8e7-207bde2ca262 ; } ; ]
2023-03-21 15:26:41 Do transfer
2023-03-21 15:26:41 Executing JS command: CallSIP with params [{id = EnXfkzRvSheoIzOY3YMOjIUBS-FzdUnAt4ZlHcW7C-E ; } ;
Then there is some sip logging.
Then it fails because of the wrong password:
2023-03-21 15:26:41 Sent event to JS onPhoneEvent with params [{code = 407 ; headers = {Reason = ; } ; id = EnXfkzRvSheoIzOY3YMOjIUBS-FzdUnAt4ZlHcW7C-E ; name = Call.Failed ; reason = Proxy Authentication Required ; } ; ]
2023-03-21 15:26:41 Transfer failed: {"code":407,"headers":{"Reason":""},"id":"EnXfkzRvSheoIzOY3YMOjIUBS-FzdUnAt4ZlHcW7C-E","name":"Call.Failed","reason":"Proxy Authentication Required","eventSourceField":"call","call":{"rism":{},"is_conf":false}}
2023-03-21 15:26:41 Executing JS command: CcaiParticipantAnalyzeContent with params [{conversationId = Le3FtBGxRIeeijnq6W0EVYg5dRYGREIql9abd_2WRbM ; id = L6rtn7f8QUqRtzHQuWVy2i4OUdglnkaRq7sollbMVew ; } ; {eventInput = {languageCode = nl ; name = TRANSFER_FAIL ; } ; } ; ]
2023-03-21 15:26:44 Sent event to JS onPhoneEvent with params [{cost = 0.00469 ; direction = All numbers ; duration = 24 ; headers = {} ; id = A111E0CD26143ED7.1679412381.1949363 ; internalCode = 200 ; name = Call.Disconnected ; reason = Normal call clearing ; } ; ]
It looks like StopMedia is called last before the TRANSFER_FAIL event is triggert.
Thanks for the help.
Nahh ive tried some more
conversationParticipant.addEventListener(CCAI.Events.Participant.PlaybackFinished, (e) => {
if (transfer) {
transfer = false;
/* callPSTN calls the first phone number and gets the phone number dynamic from the incoming call from call.callerid()
The first phone number has to be verified by Voximplant/Settings/CallerID */
outboundCall = VoxEngine.callPSTN('+0000000000', call.callerid());
Logger.write("executed calllPSTN")
outboundCall.addEventListener(CallEvents.Connected, () => {
Logger.write("call connected")
success = true;
conversationParticipant.addPlaybackMarker(0);
endConversation();
});
outboundCall.addEventListener(CallEvents.Failed, (e) => {
Logger.write(`Transfer failed: ${JSON.stringify(e)}`)
failure = true;
conversationParticipant.addPlaybackMarker(0);
});
}
else if (hangup) {
endConversation();
}
});
conversationParticipant.addEventListener(CCAI.Events.Participant.MarkerReached, (e) => {
Logger.write("MARKER REACHED IN USER VE SCRIPT")
if(success){
call.stopMediaTo(conversationParticipant);
conversationParticipant.analyzeContent({
eventInput: { name: "TRANSFER_SUCCESS", languageCode: languageCode },
});
endConversation();
}else if(failure){
call.stopMediaTo(conversationParticipant);
conversationParticipant.analyzeContent({
eventInput: { name: "TRANSFER_FAIL", languageCode: languageCode },
});
}
});
after the transfer fails the addPlaybackmarker gets executed but the MarkerReached event does not trigger and the Event is not sent.
023-03-22 07:57:36 Executing JS command: CallPSTN with params [{id = cdspU_FMShK38PqhGRtioYFobPsLzkxKlKyQ4UcADiM ; } ; {callerid = 41798524132 ; carrier = 0 ; dump = NULL ; number = 49151100064540 ; retryOn480 = NULL ; } ; ]
2023-03-22 07:57:36 Executing JS command: EnableMediaStatistics with params [{id = cdspU_FMShK38PqhGRtioYFobPsLzkxKlKyQ4UcADiM ; } ; ]
2023-03-22 07:57:36 executed calllPSTN
2023-03-22 07:57:37 Sent event to JS onPhoneEvent with params [{code = 404 ; headers = {Reason = Q.850;cause=1;text="Unallocated";iintcode=12013;isubsystem=9 ; } ; id = cdspU_FMShK38PqhGRtioYFobPsLzkxKlKyQ4UcADiM ; name = Call.Failed ; reason = Not Found ; } ; ]
2023-03-22 07:57:37 Transfer failed: {"code":404,"headers":{"Reason":"Q.850;cause=1;text=\"Unallocated\";iintcode=12013;isubsystem=9"},"id":"cdspU_FMShK38PqhGRtioYFobPsLzkxKlKyQ4UcADiM","name":"Call.Failed","reason":"Not Found","eventSourceField":"call","call":{"rism":{},"is_conf":false}}
2023-03-22 07:57:37 Executing JS command: CcaiParticipantAddPlaybackMarker with params [{conversationId = lGadmzhISE28nfkFgWkpPUcCIIP7GEZGuQssG6IVz50 ; id = O17kqKwIQnuiTy7Q1A9-gz6RfVEo3knSlcgJCedMb6Q ; } ; {offset = 0 ; } ; ]
2023-03-22 07:57:48 Sent event to JS onPhoneEvent with params [{cost = 0.004655 ; direction = All numbers ; duration = 49 ; headers = {} ; id = 0361B6B046342332.1679471819.2340406 ; internalCode = 200 ; name = Call.Disconnected ; reason = Normal call clearing ; } ; ]
2023-03-22 07:57:48 Executing JS command: CcaiCompleteConversation with params [{id = lGadmzhISE28nfkFgWkpPUcCIIP7GEZGuQssG6IVz50 ; } ; ]
all you need to do is to have this code in your scenario:
conversationParticipant.addEventListener(CCAI.Events.Participant.MarkerReached, (e) => {
Logger.write(“MARKER REACHED IN USER VE SCRIPT”)
call.stopMediaTo(conversationParticipant);
})
It will stop sending media to Dialogflow from the call to DF, after that you can send either event to the DF using analyzeContent, or media using startMediaTo.
Thatis exactly what I have in my code, but it looks like the analizeContent doesn’t work.
conversationParticipant.analyzeContent({
eventInput: { name: "TRANSFER_FAIL", languageCode: languageCode },
});
I’ve tried everything but unfortunately it just doesn’t work.
It does work for me, so should work for you as well. Hard to tell without seeing the whole scenario code where the issue is, you can remove all private info / credentials from your scenario and post it here, then we will check and let you know
The human handover itself does work, but the events “TRANSFER_FAIL” and “TRANSFER_SUCCESS” do never arrive at Dialogflow. I try to hit the TRANSFER_FAIL by using a wrong password.
This is the complete code:
require(Modules.AI);
const languageCode = "";
const agentId = 0000;
const profile = "";
const appName = "";
const region = "";
let agent,
call,
conversation,
conversationParticipant,
isConversationCreated = false,
isCallCreated = false,
isCallConnected = false,
isParticipantCreated = false,
hangup = false,
transfer = false,
outboundCall,
phonenumberCaller,
phonenumberCalled,
singleResponse = false,
sipUsername,
sipPassword,
sipConnectionstring,
shortAnswers = []; //["ja","nee"];
VoxEngine.addEventListener(AppEvents.Started, function(ev) {
agent = new CCAI.Agent(agentId, region);
agent.addEventListener(CCAI.Events.Agent.Started, () => {
conversation = new CCAI.Conversation({
agent,
profile: {
name: profile
},
project: appName
});
conversation.addEventListener(CCAI.Events.Conversation.Created, () => {
isConversationCreated = true;
createParticipant();
});
});
});
VoxEngine.addEventListener(AppEvents.CallAlerting, function(ev) {
isCallCreated = true;
createParticipant();
call = ev.call;
call.record();
phonenumberCaller = ev.callerid;
displayName = ev.displayName;
if(ev.headers.Diversion != null){
phonenumberCalled = ev.headers.Diversion.substring(ev.headers.Diversion.indexOf(':')+1,ev.headers.Diversion.indexOf('@'));
}else {
phonenumberCalled = ev.destination;
}
call.answer();
call.addEventListener(CallEvents.Connected, function() {
isCallConnected = true;
});
call.addEventListener(CallEvents.Disconnected, function() {
conversationParticipant.analyzeContent({
eventInput: { name: "EARLY_HANGUP", languageCode: languageCode },
});
conversation.stop();
VoxEngine.terminate();
});
});
function endConversation() {
conversation.stop();
call.hangup();
VoxEngine.terminate();
}
function createParticipant() {
if (!isConversationCreated || !isCallCreated) return;
conversationParticipant = conversation.addParticipant({
call: call,
options: {
role: "END_USER"
},
dialogflowSettings: {
enableMixingAudio: true,
lang: languageCode,
singleUtterance: true,
replyAudioConfig: {
audioEncoding: "OUTPUT_AUDIO_ENCODING_OGG_OPUS",
// Synthesized voice configuration
synthesizeSpeechConfig: {
voice: {
name: "nl-NL-Wavenet-E"
}
},
}
}
});
conversationParticipant.addEventListener(CCAI.Events.Participant.Created, () => {
isParticipantCreated = true;
setupMedia();
});
conversationParticipant.addEventListener(CCAI.Events.Participant.Response, (e) => {
if (e.response.automatedAgentReply?.responseMessages) {
e.response.automatedAgentReply.responseMessages.forEach((response) => {
if (response.liveAgentHandoff) transfer = true;
if (response.endInteraction && e.response.replyText) hangup = true;
else if (response.endInteraction) endConversation();
})
}
});
conversationParticipant.addEventListener(CCAI.Events.Participant.Response, (e) => {
if (e.response.automatedAgentReply) {
let webhookPayload = e.response.automatedAgentReply.parameters
if (
webhookPayload &&
webhookPayload.expectShortAnswer &&
webhookPayload.shortAnswers &&
webhookPayload.shortAnswers.length > 0
) {
Logger.write("Expecting a short answer, so starting check for intermediary speech recognition results");
singleResponse = true;
shortAnswers = webhookPayload.shortAnswers;
Logger.write(shortAnswers);
} else {
singleResponse = false;
shortAnswers = [];
}
}
if (!singleResponse && e.response.automatedAgentReply?.responseMessages) {
e.response.automatedAgentReply.responseMessages.forEach((response) => {
if (response.liveAgentHandoff) {
Logger.write("Agent handoff");
let webhookPayload = e.response.automatedAgentReply.parameters;
sipUsername = webhookPayload.SIP_USERNAME;
sipPassword = webhookPayload.SIP_PASSWORD;
sipConnectionstring = webhookPayload.SIP_CONNECTIONSTRING;
transfer = true;
}
if (response.endInteraction && e.response.replyText) {
Logger.write("Hangup");
hangup = true;
}
else if (response.endInteraction) {
Logger.write("End conversation");
endConversation();
}
})
}else{
if (singleResponse && e.response.recognitionResult && !e.response.automatedAgentReply) {
const transcript = e.response.recognitionResult.transcript;
if (transcript) {
Logger.write("Found transcript");
if (e.response.recognitionResult.isFinal) {
Logger.write("Got a final response from Dialogflow so sending query")
call.stopMediaTo(conversationParticipant) // Stop sending media to Dialogflow
} else if (singleResponse) {
if (shortAnswers.includes(transcript.toLowerCase())) {
Logger.write("Short answer found");
singleResponse = false;
call.stopMediaTo(conversationParticipant);
}
else {
Logger.write("No short answer match");
call.stopMediaTo(conversationParticipant);
}
}
} else {
Logger.write("No matches");
call.stopMediaTo(conversationParticipant);
}
}
}
});
conversationParticipant.addEventListener(CCAI.Events.Participant.PlaybackFinished, (e) => {
if (transfer) {
Logger.write("Do transfer");
transfer = false;
outboundCall = VoxEngine.callSIP(sipConnectionstring, {
authUser: sipUsername,
password: "wrong_password",
extraHeaders: {},
video: false,
outProxy: null
});
outboundCall.addEventListener(CallEvents.Connected, () => {
Logger.write('Transfer connected');
VoxEngine.easyProcess(call, outboundCall, () => {
conversationParticipant.analyzeContent({
eventInput: { name: "TRANSFER_SUCCESS", languageCode: languageCode, parameters:{} },
});
endConversation();
});
});
outboundCall.addEventListener(CallEvents.Failed, (a) => {
Logger.write('Transfer failed')
conversationParticipant.analyzeContent({
eventInput: { name: "TRANSFER_FAIL", languageCode: languageCode, parameters:{} },
});
});
}else
if (hangup) {
endConversation();
}
})
conversationParticipant.addEventListener(CCAI.Events.Participant.MarkerReached, (e) => {
// without transfer check, the conversation stops immediately
if (transfer) {
Logger.write("MARKER REACHED IN USER VE SCRIPT")
call.stopMediaTo(conversationParticipant);
}
});
}
function setupMedia() {
if (!isParticipantCreated || !isCallConnected) return;
conversationParticipant.analyzeContent({
eventInput: {
name: "WELCOME",
languageCode: languageCode,
parameters: {
phonenumber: phonenumberCaller,
phonenumberCalled: phonenumberCalled
}
},
});
conversationParticipant.sendMediaTo(call);
}
Thanks for the help.
Greetings
Do you have a message for playback in case of the live agent handoff response that is played back to the user? Please send me the call log using Direct Message functionality of the forum
Hi, I checked your code and enabled Cloud logging for the DF agent, I see that events are actually reaching DF backend in Logs section of Google Cloud console. Try to enable cloud logging and give it a try.