Ask ChatGPT Vision action

DAP action

Last published at: 2024-09-10 14:38:06 UTC
Delete

The Ask ChatGPT Vision action is available for the Premium and PDM editions.

Use the Ask ChatGPT Vision action to extend the use of artificial intelligence in your daily operations even further. Let ChatGPT analyze images attached to your records, and apply its conclusions to your workflows. For example, extract details such as errors or invoice items from images, or generate text descriptions of images.
This action can be used in combination with ChatGPT models that accept image input.

Use the action via the action launcher to get suggestions from ChatGPT for your specific questions related to the records you selected, or create macros with preconfigured prompts and reply-fields for repeated use in batch jobs and flows, or by your co-workers. Apply field variables from the object and related Salesforce objects to integrate actual Salesforce data into the dialogue with ChatGPT, and to get different answers for each record. Store the responses in your Salesforce records for further processing. 

The Ask ChatGPT Vision action is available for all Salesforce objects that can contain attachments. A couple of prompt examples and use cases are listed below.

Prerequisites

Before you can start using the Ask ChatGPT Vision action, make sure you have done the following:

  • Subscribe to OpenAI's ChatGPT service at https://openai.com/pricing. The Ask ChatGPT Vision action can use text-based models that also accept image input, such as GPT-4o and GPT-4 Turbo.
  • Enable remote site settings to allow Salesforce to call out to ChatGPT:
    1. Go to Salesforce Setup > Settings: Security > Remote Site Settings. 
    2. Click New Remote Site.
    3. Enter a name for the new remote site.
    4. At Remote Site URL, enter https://api.openai.com 
    5. Make sure the Active box is ticked. 
    6. Click Save.
  • Set up the Ask ChatGPT action settings:
    1. Go to DAP Setup. At left, go to the Action Library. At the Ask ChatGPT Vision action, click Edit.
    2. At OpenAI API Key, enter the API Key belonging to your OpenAI subscription.
    3. At OpenAI Model, enter the API name of the OpenAI model you want to use.
      For example, enter gpt-4o or gpt-4-turbo.
    4. At OpenAI Detail Mode, enter the detail mode to use: auto, low, or high. Low resolution mode returns responses faster, and uses fewer input tokens. Read more about the detail parameter here.
    5. At OpenAI Temperature, enter a sampling temperature value between 0 and 2. Higher values like 1.5 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.

Image Requirements

  • The Ask ChatGPT Vision action can process image files of the types PNG, JPG, WEBP, and non-animated GIF.
  • The maximum file size is 2MB when not using a public link.
    • To analyze larger images, create a public link for the image file. This will also increase processing speed. Just creating the public link is sufficient, there's no need to place the link anywhere in particular. Do note that anyone who can access the public link can also access the file it links to.
    • Note that public links cannot be created when Content Distribution is disabled. In that case only files of 2MB or smaller can be processed. However, in most Production Orgs, Content Distribution is enabled.
Delete

Defining the right prompt

ChatGPT is a chatbot that will usually answer in full sentences. To have it return text that you can use in your records, you will have to tell it specifically what kind of answer you want. For example, if you want it to return one of the values of a picklist, add a sentence to the prompt listing all values of the picklist and say that it should only return one of those values. It's best to try it out with a text field first, to see what kind of answers you will get.

Add Field Variables to your prompt to use specific data from each record. See the examples below.

Note that you can enter a long prompt, but Salesforce field limits do apply.

Using the Action

  1. Select the records that you want to apply the action to in a list view, or by creating a new job.
  2. In the Action Launcher (top right in list view, or the next window in the DAP Job modal), select Ask ChatGPT Vision and click Next.
  3. In the next window, fill in the fields:
    1. At Prompt, fill in the prompt you want to use. 
    2. If desired, add field variables: place the cursor at the right spot in the text and click </> Field Variable  . Select the field you want to use the values of.
    3. Select an Attachment Selection Criterion plus a Fallback Rule.
      Only one attachment can be processed per record. The criterion determines which attached image will be analyzed. The fallback rule selects an image if two or more files meet the criterion.
      One of the criterion options is Attachment File Name. If selected, enter an Operator and Value (e.g. "Contains value: Screenshot") to determine the file name of the image you want to have analyzed.
    4. At Result Field, select a text field or picklist where you want to record the answers returned by ChatGPT. 
    5. The Maximum Token Response Size shows the maximum output token size based on the size of the selected Result Field. The size set here makes sure ChatGPT does not give an answer that is larger than would fit in the result field. A larger token response size would unnecessarily use up extra tokens, as the last part of larger answers cannot be stored in the result field. You can set a smaller token response size to get smaller answers and reduce cost.
      Read more about tokens here.
    6. Click Next.
      The configuration for an Ask ChatGPT Vision action with the field variable for the Description field used in the prompt. The result field is a custom picklist field with values Yes and No.
  4. In the confirmation window, check the details for the action, and click Start.
  5. If you ran the action from a list view: once the progress bar is at 100% completed, click Finish to return to the object overview.
    If you ran the action as a Job, you are returned to the Job Overview, where you can view the job results information via the Info button.

The answers that ChatGPT gave to your prompt and image analysis are now saved on each record.

Delete

The Record Audit Log, if enabled, does not store the actual prompt and answer because of size limitations. It only records whether an answer was given or not.

Delete

Create a Macro

To prevent having to type out the same prompt repeatedly, to use the action in batch jobs and flows, or to have your colleagues use the Ask ChatGPT Vision action without them having to come up with a good prompt themselves, you can create a Macro‍. Assign the macro to users or profiles, and limit the use of the Ask ChatGPT Vision action itself, to make sure ChatGPT is only used for certain prompts and result fields.

Examples

Object Prompt Result Field
Case Please determine whether the image content matches this description: "{!Description}". Answer only "Yes" or "No". Custom picklist field 'Attachment matches description' with values Yes and No

Use Cases

An example of the use of Ask ChatGPT Vision action is to check whether attachments of Cases are in line with the cases' subject and description. If an attached screenshot in a case matches the content then automatically send the case to a customer support agent; if not then flag it for somebody to take a look at it first.

Or let ChatGPT Vision read restaurant receipts and create expense reports; let it examine a damage report image and have it create or check the damage report; etc etc.

Use your own classes for even more options

The Ask ChatGPT Vision action can do a lot, but it is limited to the Object fields to retrieve information from. For example, for Cases you can only use information from fields such as the Description to use in questions for ChatGPT. If you'd like to use other data sources, such as case emails for Cases, you can build your own class and action. Create a class for DAP that calls the ChatGPT API, and include all Object data you want to provide. Then run the class you created with the Run An Apex Class action. Of course you can then also create a Macro for this action.

Or, to expand your options for using the ChatGPT answers, you can also implement the Ask ChatGPT Vision action in a class, and create or update other Objects with the responses returned by ChatGPT. Send an Email Message, create a new Task, add a note to an Opportunity, the possibilities are endless. Read more about using an action in a class: Start an Action from an Apex Class.‍ 

Delete

Using ChatGPT

For subjects related to the use of ChatGPT, such as accuracy or data privacy, we kindly refer you to the information provided by OpenAI on their website.