Google Cloud Native v0.28.0, Feb 2 23

google-native.dialogflow/v2.getEvaluation

Gets an evaluation of conversation model.

Using getEvaluation

Two invocation forms are available. The direct form accepts plain arguments and either blocks until the result value is available, or returns a Promise-wrapped result. The output form accepts Input-wrapped arguments and returns an Output-wrapped result.

function getEvaluation(args: GetEvaluationArgs, opts?: InvokeOptions): Promise<GetEvaluationResult>
function getEvaluationOutput(args: GetEvaluationOutputArgs, opts?: InvokeOptions): Output<GetEvaluationResult>

def get_evaluation(conversation_model_id: Optional[str] = None,
                   evaluation_id: Optional[str] = None,
                   location: Optional[str] = None,
                   project: Optional[str] = None,
                   opts: Optional[InvokeOptions] = None) -> GetEvaluationResult
def get_evaluation_output(conversation_model_id: Optional[pulumi.Input[str]] = None,
                   evaluation_id: Optional[pulumi.Input[str]] = None,
                   location: Optional[pulumi.Input[str]] = None,
                   project: Optional[pulumi.Input[str]] = None,
                   opts: Optional[InvokeOptions] = None) -> Output[GetEvaluationResult]

func LookupEvaluation(ctx *Context, args *LookupEvaluationArgs, opts ...InvokeOption) (*LookupEvaluationResult, error)
func LookupEvaluationOutput(ctx *Context, args *LookupEvaluationOutputArgs, opts ...InvokeOption) LookupEvaluationResultOutput

> Note: This function is named LookupEvaluation in the Go SDK.

public static class GetEvaluation 
{
    public static Task<GetEvaluationResult> InvokeAsync(GetEvaluationArgs args, InvokeOptions? opts = null)
    public static Output<GetEvaluationResult> Invoke(GetEvaluationInvokeArgs args, InvokeOptions? opts = null)
}

public static CompletableFuture<GetEvaluationResult> getEvaluation(GetEvaluationArgs args, InvokeOptions options)
// Output-based functions aren't available in Java yet

fn::invoke:
  function: google-native:dialogflow/v2:getEvaluation
  arguments:
    # arguments dictionary

The following arguments are supported:

ConversationModelId string
EvaluationId string
Location string
Project string

ConversationModelId string
EvaluationId string
Location string
Project string

conversationModelId String
evaluationId String
location String
project String

conversationModelId string
evaluationId string
location string
project string

conversation_model_id str
evaluation_id str
location str
project str

conversationModelId String
evaluationId String
location String
project String

getEvaluation Result

The following output properties are available:

CreateTime string: Creation time of this model.
DisplayName string: Optional. The display name of the model evaluation. At most 64 bytes long.
EvaluationConfig Pulumi.GoogleNative.Dialogflow.V2.Outputs.GoogleCloudDialogflowV2EvaluationConfigResponse: Optional. The configuration of the evaluation task.
Name string: The resource name of the evaluation. Format: projects//conversationModels//evaluations/
RawHumanEvalTemplateCsv string: Human eval template in csv format. It tooks real-world conversations provided through input dataset, generates example suggestions for customer to verify quality of the model. For Smart Reply, the generated csv file contains columns of Context, (Suggestions,Q1,Q2)*3, Actual reply. Context contains at most 10 latest messages in the conversation prior to the current suggestion. Q1: "Would you send it as the next message of agent?" Evaluated based on whether the suggest is appropriate to be sent by agent in current context. Q2: "Does the suggestion move the conversation closer to resolution?" Evaluated based on whether the suggestion provide solutions, or answers customer's question or collect information from customer to resolve the customer's issue. Actual reply column contains the actual agent reply sent in the context.
SmartReplyMetrics Pulumi.GoogleNative.Dialogflow.V2.Outputs.GoogleCloudDialogflowV2SmartReplyMetricsResponse: Only available when model is for smart reply.

CreateTime string: Creation time of this model.
DisplayName string: Optional. The display name of the model evaluation. At most 64 bytes long.
EvaluationConfig GoogleCloudDialogflowV2EvaluationConfigResponse: Optional. The configuration of the evaluation task.
Name string: The resource name of the evaluation. Format: projects//conversationModels//evaluations/
RawHumanEvalTemplateCsv string: Human eval template in csv format. It tooks real-world conversations provided through input dataset, generates example suggestions for customer to verify quality of the model. For Smart Reply, the generated csv file contains columns of Context, (Suggestions,Q1,Q2)*3, Actual reply. Context contains at most 10 latest messages in the conversation prior to the current suggestion. Q1: "Would you send it as the next message of agent?" Evaluated based on whether the suggest is appropriate to be sent by agent in current context. Q2: "Does the suggestion move the conversation closer to resolution?" Evaluated based on whether the suggestion provide solutions, or answers customer's question or collect information from customer to resolve the customer's issue. Actual reply column contains the actual agent reply sent in the context.
SmartReplyMetrics GoogleCloudDialogflowV2SmartReplyMetricsResponse: Only available when model is for smart reply.

createTime String: Creation time of this model.
displayName String: Optional. The display name of the model evaluation. At most 64 bytes long.
evaluationConfig GoogleCloudDialogflowV2EvaluationConfigResponse: Optional. The configuration of the evaluation task.
name String: The resource name of the evaluation. Format: projects//conversationModels//evaluations/
rawHumanEvalTemplateCsv String: Human eval template in csv format. It tooks real-world conversations provided through input dataset, generates example suggestions for customer to verify quality of the model. For Smart Reply, the generated csv file contains columns of Context, (Suggestions,Q1,Q2)*3, Actual reply. Context contains at most 10 latest messages in the conversation prior to the current suggestion. Q1: "Would you send it as the next message of agent?" Evaluated based on whether the suggest is appropriate to be sent by agent in current context. Q2: "Does the suggestion move the conversation closer to resolution?" Evaluated based on whether the suggestion provide solutions, or answers customer's question or collect information from customer to resolve the customer's issue. Actual reply column contains the actual agent reply sent in the context.
smartReplyMetrics GoogleCloudDialogflowV2SmartReplyMetricsResponse: Only available when model is for smart reply.

createTime string: Creation time of this model.
displayName string: Optional. The display name of the model evaluation. At most 64 bytes long.
evaluationConfig GoogleCloudDialogflowV2EvaluationConfigResponse: Optional. The configuration of the evaluation task.
name string: The resource name of the evaluation. Format: projects//conversationModels//evaluations/
rawHumanEvalTemplateCsv string: Human eval template in csv format. It tooks real-world conversations provided through input dataset, generates example suggestions for customer to verify quality of the model. For Smart Reply, the generated csv file contains columns of Context, (Suggestions,Q1,Q2)*3, Actual reply. Context contains at most 10 latest messages in the conversation prior to the current suggestion. Q1: "Would you send it as the next message of agent?" Evaluated based on whether the suggest is appropriate to be sent by agent in current context. Q2: "Does the suggestion move the conversation closer to resolution?" Evaluated based on whether the suggestion provide solutions, or answers customer's question or collect information from customer to resolve the customer's issue. Actual reply column contains the actual agent reply sent in the context.
smartReplyMetrics GoogleCloudDialogflowV2SmartReplyMetricsResponse: Only available when model is for smart reply.

create_time str: Creation time of this model.
display_name str: Optional. The display name of the model evaluation. At most 64 bytes long.
evaluation_config GoogleCloudDialogflowV2EvaluationConfigResponse: Optional. The configuration of the evaluation task.
name str: The resource name of the evaluation. Format: projects//conversationModels//evaluations/
raw_human_eval_template_csv str: Human eval template in csv format. It tooks real-world conversations provided through input dataset, generates example suggestions for customer to verify quality of the model. For Smart Reply, the generated csv file contains columns of Context, (Suggestions,Q1,Q2)*3, Actual reply. Context contains at most 10 latest messages in the conversation prior to the current suggestion. Q1: "Would you send it as the next message of agent?" Evaluated based on whether the suggest is appropriate to be sent by agent in current context. Q2: "Does the suggestion move the conversation closer to resolution?" Evaluated based on whether the suggestion provide solutions, or answers customer's question or collect information from customer to resolve the customer's issue. Actual reply column contains the actual agent reply sent in the context.
smart_reply_metrics GoogleCloudDialogflowV2SmartReplyMetricsResponse: Only available when model is for smart reply.

createTime String: Creation time of this model.
displayName String: Optional. The display name of the model evaluation. At most 64 bytes long.
evaluationConfig Property Map: Optional. The configuration of the evaluation task.
name String: The resource name of the evaluation. Format: projects//conversationModels//evaluations/
rawHumanEvalTemplateCsv String: Human eval template in csv format. It tooks real-world conversations provided through input dataset, generates example suggestions for customer to verify quality of the model. For Smart Reply, the generated csv file contains columns of Context, (Suggestions,Q1,Q2)*3, Actual reply. Context contains at most 10 latest messages in the conversation prior to the current suggestion. Q1: "Would you send it as the next message of agent?" Evaluated based on whether the suggest is appropriate to be sent by agent in current context. Q2: "Does the suggestion move the conversation closer to resolution?" Evaluated based on whether the suggestion provide solutions, or answers customer's question or collect information from customer to resolve the customer's issue. Actual reply column contains the actual agent reply sent in the context.
smartReplyMetrics Property Map: Only available when model is for smart reply.

Supporting Types

GoogleCloudDialogflowV2EvaluationConfigResponse

Datasets List<Pulumi.GoogleNative.Dialogflow.V2.Inputs.GoogleCloudDialogflowV2InputDatasetResponse>: Datasets used for evaluation.
SmartComposeConfig Pulumi.GoogleNative.Dialogflow.V2.Inputs.GoogleCloudDialogflowV2EvaluationConfigSmartComposeConfigResponse: Configuration for smart compose model evalution.
SmartReplyConfig Pulumi.GoogleNative.Dialogflow.V2.Inputs.GoogleCloudDialogflowV2EvaluationConfigSmartReplyConfigResponse: Configuration for smart reply model evalution.

Datasets []GoogleCloudDialogflowV2InputDatasetResponse: Datasets used for evaluation.
SmartComposeConfig GoogleCloudDialogflowV2EvaluationConfigSmartComposeConfigResponse: Configuration for smart compose model evalution.
SmartReplyConfig GoogleCloudDialogflowV2EvaluationConfigSmartReplyConfigResponse: Configuration for smart reply model evalution.

datasets List<GoogleCloudDialogflowV2InputDatasetResponse>: Datasets used for evaluation.
smartComposeConfig GoogleCloudDialogflowV2EvaluationConfigSmartComposeConfigResponse: Configuration for smart compose model evalution.
smartReplyConfig GoogleCloudDialogflowV2EvaluationConfigSmartReplyConfigResponse: Configuration for smart reply model evalution.

datasets GoogleCloudDialogflowV2InputDatasetResponse[]: Datasets used for evaluation.
smartComposeConfig GoogleCloudDialogflowV2EvaluationConfigSmartComposeConfigResponse: Configuration for smart compose model evalution.
smartReplyConfig GoogleCloudDialogflowV2EvaluationConfigSmartReplyConfigResponse: Configuration for smart reply model evalution.

datasets Sequence[GoogleCloudDialogflowV2InputDatasetResponse]: Datasets used for evaluation.
smart_compose_config GoogleCloudDialogflowV2EvaluationConfigSmartComposeConfigResponse: Configuration for smart compose model evalution.
smart_reply_config GoogleCloudDialogflowV2EvaluationConfigSmartReplyConfigResponse: Configuration for smart reply model evalution.

datasets List<Property Map>: Datasets used for evaluation.
smartComposeConfig Property Map: Configuration for smart compose model evalution.
smartReplyConfig Property Map: Configuration for smart reply model evalution.

GoogleCloudDialogflowV2EvaluationConfigSmartComposeConfigResponse

AllowlistDocument string: The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.
MaxResultCount int: The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

AllowlistDocument string: The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.
MaxResultCount int: The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument String: The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.
maxResultCount Integer: The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument string: The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.
maxResultCount number: The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlist_document str: The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.
max_result_count int: The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument String: The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart compose model.
maxResultCount Number: The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

GoogleCloudDialogflowV2EvaluationConfigSmartReplyConfigResponse

AllowlistDocument string: The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.
MaxResultCount int: The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

AllowlistDocument string: The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.
MaxResultCount int: The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument String: The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.
maxResultCount Integer: The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument string: The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.
maxResultCount number: The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlist_document str: The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.
max_result_count int: The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

allowlistDocument String: The allowlist document resource name. Format: projects//knowledgeBases//documents/. Only used for smart reply model.
maxResultCount Number: The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.

GoogleCloudDialogflowV2InputDatasetResponse

Dataset string: ConversationDataset resource name. Format: projects//locations//conversationDatasets/

Dataset string: ConversationDataset resource name. Format: projects//locations//conversationDatasets/

dataset String: ConversationDataset resource name. Format: projects//locations//conversationDatasets/

dataset string: ConversationDataset resource name. Format: projects//locations//conversationDatasets/

dataset str: ConversationDataset resource name. Format: projects//locations//conversationDatasets/

dataset String: ConversationDataset resource name. Format: projects//locations//conversationDatasets/

GoogleCloudDialogflowV2SmartReplyMetricsResponse

AllowlistCoverage double: Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].
ConversationCount string: Total number of conversations used to generate this metric.
TopNMetrics List<Pulumi.GoogleNative.Dialogflow.V2.Inputs.GoogleCloudDialogflowV2SmartReplyMetricsTopNMetricsResponse>: Metrics of top n smart replies, sorted by TopNMetric.n.

AllowlistCoverage float64: Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].
ConversationCount string: Total number of conversations used to generate this metric.
TopNMetrics []GoogleCloudDialogflowV2SmartReplyMetricsTopNMetricsResponse: Metrics of top n smart replies, sorted by TopNMetric.n.

allowlistCoverage Double: Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].
conversationCount String: Total number of conversations used to generate this metric.
topNMetrics List<GoogleCloudDialogflowV2SmartReplyMetricsTopNMetricsResponse>: Metrics of top n smart replies, sorted by TopNMetric.n.

allowlistCoverage number: Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].
conversationCount string: Total number of conversations used to generate this metric.
topNMetrics GoogleCloudDialogflowV2SmartReplyMetricsTopNMetricsResponse[]: Metrics of top n smart replies, sorted by TopNMetric.n.

allowlist_coverage float: Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].
conversation_count str: Total number of conversations used to generate this metric.
top_n_metrics Sequence[GoogleCloudDialogflowV2SmartReplyMetricsTopNMetricsResponse]: Metrics of top n smart replies, sorted by TopNMetric.n.

allowlistCoverage Number: Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].
conversationCount String: Total number of conversations used to generate this metric.
topNMetrics List<Property Map>: Metrics of top n smart replies, sorted by TopNMetric.n.

GoogleCloudDialogflowV2SmartReplyMetricsTopNMetricsResponse

N int: Number of retrieved smart replies. For example, when n is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model.
Recall double: Defined as number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply divided by number of queries with at least one smart reply. Value ranges from 0.0 to 1.0 inclusive.

N int: Number of retrieved smart replies. For example, when n is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model.
Recall float64: Defined as number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply divided by number of queries with at least one smart reply. Value ranges from 0.0 to 1.0 inclusive.

n Integer: Number of retrieved smart replies. For example, when n is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model.
recall Double: Defined as number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply divided by number of queries with at least one smart reply. Value ranges from 0.0 to 1.0 inclusive.

n number: Number of retrieved smart replies. For example, when n is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model.
recall number: Defined as number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply divided by number of queries with at least one smart reply. Value ranges from 0.0 to 1.0 inclusive.

n int: Number of retrieved smart replies. For example, when n is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model.
recall float: Defined as number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply divided by number of queries with at least one smart reply. Value ranges from 0.0 to 1.0 inclusive.

n Number: Number of retrieved smart replies. For example, when n is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model.
recall Number: Defined as number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply divided by number of queries with at least one smart reply. Value ranges from 0.0 to 1.0 inclusive.

Package Details

Repository: Google Cloud Native pulumi/pulumi-google-native
License: Apache-2.0