Massive Multimodal Models

  • Code computer programs and check for bugs in code.
  • Compose music.
  • Draft emails.
  • Summarize articles, podcasts or presentations.
  • Script social media posts.
  • Create titles for articles.
  • Solve math problems.
  • Discover keywords for search engine optimization.
  • Create articles, blog posts and quizzes for websites.
  • Reword existing content for a different medium, such as a presentation transcript for a blog post.
  • Formulate product descriptions.
  • Play games.
  • Assist with job searches, including writing resumes and cover letters.
  • Ask trivia questions.
  • Describe complex topics more simply.
  • Write video scripts.
  • Research markets for products.
  • Generate art." [04] Amanda Hetler

By Bing Image Creator

We can’t ignore the impact on our day-to-day activities of AI tools. They make the communication between humans and machines easier and simpler.

We can quickly have new insights and information on different fields of interests. But in order to improve the quality and the accuracy of responses we have to structure and phrase of the prompt appropriately.

It’s important to remember that ChatGPT doesn’t understand and doesn’t think, it generates responses based on patterns it learned during training.

The engine of ChatGPT is based on the concept of “token”. GPT (Generative Pre-trained Transformer) model generates the tokens predicting the most probable subsequent token using complex linear algebra.

The model uses an iterative process. It generates one token at time and after generating each token, it revisits the entire sequence of generated tokes and processes them again to generate the next token.

The prompt engineering is very important for creating better AI-powered services and obtaining useful results from AI Tools.

When you craft the prompt, it’s important to bear in mind that ChatGPT has a token limit (generally 2048 tokens), which include both the prompt and the generated response. Long prompts can limit the length of response, for this reason it is important to keep prompts concise.

Let us now analyze some techniques of prompt engineering.

It is used to gather information and to answer question what and how. Examples of prompt:

  • What are the best restaurants in Rome?
  • How do I cook pizza?

The prompts provide information to the model to perform a specific task. Example:

Prompt: I am planning to celebrate the Italian Republic Day in the [COUNTRY] can you suggest some original ideas to make it more enjoyable?

It is used to ask the model to compare and to evaluate different options to help the user make an appropriate decision. Example:

Prompt: What are the strengths and weaknesses of [Option A] compared to [Option B].

It is used to ask the model to get the AI’s opinion on a given topic. Example:

Prompt: What would happen if we use only public transport in Rome?.

If you ask the model for generic question, you receive generic answer. You have to define your prompt with clear instruction and precise and descriptive information.

If you want to have a specific output or format of the output from ChatGPT, for example a program in Python or Visual Foxpro.

We try with a simple e specific request.

Prompt: Write a function in Python that takes as input three integers and gives as output the maximum of these three numbers.

ChatGPT 3.5 answer:

Prompt: Write a function in Visual Foxpro that takes as input three integers and gives as output the maximum of these three numbers.

Amazing, ChatGPT 3.5 answer:

VERY IMPORTANT

AI-generated code may need to be modified or tested before deploying it. 
It is strongly recommended to:
- always modify and review the generated code to ensure it meets your specific requirements;
- use it as STARTING-POINT;
- test and check the code;
- it ALWAYS NEEDS HUMAN OVERSIGHT.

You can also act as someone else when you interact with ChatGPT.

If you add a role to the question, you make ChatGPT change the answer and the quality and the tone of the output. In this way we got much better information. Here it is a schema to use when you build a prompt related a role playing:

  1. WHO: you can ask ChatGPT to be what you want. You assign the role you need the model to play. A scientist, doctor, business man, chef and so on.
  2. WHEN: you can put the character at any moment in time;
  3. WHERE: you can put the character to a particular location or context.
  4. WHY: you want to dialogue with the character for whatever reason, motivations or purpose you want;
  5. WHAT: you want to dialogue with the character about what. That is the action you want the model to do.

We just need to verify the level of reliability and credibility given to this type of interaction.

Here is some practical examples.

Act as a character from a book:

Prompt: I want you to act like [character] from [book]. I want you to respond and answer like [character] using the tone, manner and vocabulary [character] would use.

Act as historical character:

Prompt: I want you to act as [historical character] to better understand the historical facts of that period.

Act as a political character:

Prompt: I want you to act as [political character] in order to ask as improving the quality of life of the people.

Act as a scientist:

Prompt: I want you to act as a scientist. You will apply your knowledge of scientific to propose useful strategies for saving the environment from pollution.

Act as a travel guide:

Prompt: I want you to act as a travel guide from Italy at the time of the Roman empire when Caesar was emperor. I will write you my location and you will suggest a place to visit near my location.

In zero-shot prompting, we use a prompt that describes the task, but it doesn't contain examples or demonstrations.

You use this prompt when you trust the model’s knowledge to provide a sufficient answer.

Prompt: Write a description of the Colosseum.

It involves providing the model a few examples to guide its understanding of the desired outcome.

The example will be of:

  • Knowledge extracting;
  • And it’s formatting.

We can define the prompt like this:

Prompt: Here are some examples of each item of the list of best important business people.

  • X is the Y of Z
  • X -> [PERSON]
  • Y -> [POSITION/TITLE]
  • Z -> [COMPANY]

You can also combine all these techniques:

  • Directional prompting;
  • Output formatting;
  • Role based prompting;
  • Few shots prompting.

This technique encourages the model to break down complex tasks into smaller intermediate steps before arriving to conclusion. It improves the multi-step reasoning abilities of large language models (LLMs) and is helpful for complex problems that would be difficult or impossible to solve in a single step.

There are also variants of CoT prompting, such as "Tree-of-Thought" and "Graph-of-Thought", which were inspired by the success of CoT prompting.

You can ask the model for the style of the output:

  1. Writing as another author;
  2. As emotional state;
  3. In enthusiastic tone;
  4. Writing something in a sad state;
  5. Rewriting the following email in my style of writing;
  6. Rewriting the following email in the style of xy;

You can ask the model to extract information in the way is useful. You can ask to extract information from the example and structure it into markdown table or a specific format.

Prompt: Generate a table of three column: name, function, phone numbers from the text.

Text: “Urs Wiedmer
Head of Communications
+41584645082
+41796919559
Markus Spörndli
Press spokesperson
Deputy Head of Communications
+41584634149
+41796747396
Irène Harnischberg
Press spokesperson
+41584622034
+41794567139
Charles-Étienne Viladoms
Webmaster
+41584622054
+41792194031
Loïc Zen-Ruffinen
Social Media Manager
Press spokesperson for French-speaking Switzerland
+41584817911
+41791507632"

Source: “https://www.wbf.admin.ch/wbf/en/home/dokumentation/dienstleistungen/dienstleistungen-wbf/zugang-zu-amtlichen-dokumenten.htm

You can give to the model a big chunk of text and ask to summarize it.

For example, You can prompt:

Prompt: You are summarization bot any text that I provide to you summarize it, and create a title from it.

But a more effective technique could be:

Prompt: summarize the text below as a bullet point list of the the most import points.

Text: “ … “

If you need to generate a brief overview of a scientific paper don’t use generic instruction like “summarize the scientific paper” instead you should be more specific.

Prompt: generate a brief (approx. 300 words), of the following scientific paper. The summary should be understandable and clear especially to someone with no scientific background.

Paper: “ … “

You can use the model:

  • As SPAM DETECTOR in the mail;
  • To perform SENTIMENT ANALYSIS for brands and so on.

You can prompt:

You are a sentiment analysis bot. Classify any text that I provide into three classes:

  1. NEGATIVE
  2. POSITIVE
  3. NEUTRAL

The AI-generated responses aren’t always correct. You have always to verify that the AI-generated output is accurate and up-to-date. This is important of you want to make an informed decision based on the response generated.

In any case, it is a good practice to have some idea of what you are asking for in order to properly evaluate the answer obtained from AI.

[01] openai.com;

[02] “ChatGPT Teacher Tips Part 1: Role-Playing Activities” https://edtechteacher.org/chatgptroleplaying/, March 2023;

[03] Aayush Mittal, “The Essential Guide to Prompt Engineering”, https://www.unite.ai/prompt-engineering-in-chatgpt/, April 2024;

[04] Amanda Hetler, “ChatGPT”, https://www.techtarget.com/whatis/definition/ChatGPT, December 2023;

{[(homo scripsit)]} - Not generated by AI tools or platforms.

LLMs - Large Language Models

LLMs are neural networks that can process and generate natural language text.

Midjourney Bot
APP
 — Today at 19:22
LLMs are neural networks that can process and generate natural language text. - Image #1 @Ahdpea8
Midjourney Bot APP — Today at 19:22 LLMs are neural networks that can process and generate natural language text. - Image #1 @Ahdpea8

TRAINING PHASE

They are trained on a dataset of billions of sentences using unsupervised learning techniques. In the training process LLMs learn what is the most likely word to came next to the previous one based on huge amount of data.

INPUT BY USER

LLMs accept as input a text prompt by a user and in relation with it generate in output text, word by word (token by token).

GENERATION OF THE OUTPUT

The generation process consists in predicting the next word on the base of previously generated words. LLMs are trained in doing this without any consciousness which is a prerogative of the human mind.

In this example we use as data the dystopian novel “Nineteen Eighty-Four – 1984” by English writer George Orwell, published on 1949.

Using the text of the novel as a data source, the following tables were produced. I show only a part of them:

Word/TokenOccurrences
the6249
of3309
a2482
and2326
to2236
was2213
He1959
It1864
in1759
that1457
had1311
his1079
you1011
not827
with771
as672
At654
they642
for615
IS614
but611
be608
on604
were583
there559
Winston526
him512
i495
which443
s439
one426
or424
Word/TokenWord NextScoreProbability
ofthe7430,01139
Itwas5890,00903
inthe5740,00880
Hehad3550,00544
hewas2730,00418
onthe2300,00352
wasa2250,00345
therewas2230,00342
tothe2120,00325
OBrien2050,00314
tobe2030,00311
andthe2030,00311
hadbeen2020,00310
theparty1950,00299
atthe1830,00280
thathe1670,00256
fromthe1610,00247
witha1580,00242
didnot1480,00227
thatthe1470,00225
ofa1450,00222
ofhis1450,00222
outof1420,00218
wasnot1300,00199
withthe1270,00195
hecould1240,00190
itis1240,00190
inhis1230,00188
ina1220,00187
Theywere1220,00187
seemedto1150,00176
wasthe1100,00169
couldnot1090,00167
hesaid1090,00167
thesame1030,00158
forthe1010,00155
bythe950,00146
fora920,00141
intothe920,00141
shehad870,00133
asthough820,00126
theyhad800,00123
thatit800,00123
havebeen790,00121
anda780,00120
ithad770,00118
Theother760,00116
ofthem760,00116
tohim750,00115
thetelescreen750,00115
BIGBROTHER730,00112

This is a simple diagram to understand how the text is generated word by word.

For example, if I start with BIG, LLM will probably generate BROTHER, and continuing we can produce this sentence:

BIGBROTHERwasasortofthethought
Probability0,001120,00050,003450,000950,001040,000230,00087

By using “prompt” mechanism you can ask ChatGPT for what you want using the natural language.

But how ChatGPT “UNDERSTAND” text inserted by the user?

The text is transformed and each word represented by a code that computer can processed.

A way to represent individual words is Word2Vec technique in natural language processing (NLP), in which each word is represented by a vector (a set of numbers). This helped a computer to assign a meaning to the word.

Word2Vec stands for “words as vectors”. It means expressing each word in your text corpus in n-dimensional space. The word’s weight in each dimension defines it for the model.

The meaning of the words is based on the context defined by its neighboring words where they are associated.

A simple example of word representation using the Word2Vec approach in two-dimensional space.

Man = [1,4]

Woman = [1,3]

Manager = [4,2]

Actress = [4,1]

Manager-Man+Woman=Actress
[4,2][1,4][1,3][4,1]


In the following picture we have the graphic representation.

This is what happens when you sent some prompt to ChatGPT.

  1. The text is converted and split in tokens;

[10,10], [10,31], [10,15], [14,44], [8,5], …

(you, are , an, ICT, specialist, with, a, lot, of, experience)

  1. An algorithm (like ChatGPT) makes some prediction and output text word by word.

[10,10],…

(you,can,have, an, important, and, well-paid, job)

Let us now analyze some techniques to better exploit the potential of ChatGPT.

DIRECTIONAL PROMPTING

If you submit the same question to ChatGPT many times, you will likely receive different answers.

How can you use directional prompting in order to get more precise answer?

You have to give more information and to be more descriptive when you define a prompt. You have to give clear instruction. This will help the model to understand of what you want. If you ask for generic question, you receive generic answer.

Generic question:

More specific question:

More contextual and specific question:

OUTPUT FORMATTING

If you want to have a specific output or format of the output from ChatGPT, for example CSV (Comma Separated Values), Microsoft Excel, Microsoft Word or simply txt or maybe code as well, you have to specify as in the following examples.

We want statistical data in CSV format:

[01] openai.com;

[02] KENNETH WARD CHURCH, Emerging Trends Word2Vec, IBM 2016;

"Artificial Intelligence attempts to coax a machine, typically a computer, to behave in ways humans judge to be intelligent" John McCarthy (1927-2011)

"I think that there is a lot of fear about robots and artificial intelligence among some people, whereas I'm more afraid of natural stupidity" Eugenia Cheng

Created with Midjourney Bot#9282

Now we can talk with a “machine” using a natural language instead of using programming languages consisting of specific words to be written following a strict syntax and form.

We have a new paradigm in the HCI (Human Computer Interaction) with generative models and large language models.

We can interact with these AI Systems by using “prompt” mechanism, in which we have flexible inputs continued by equally flexible outputs.

In [01] this mechanism is called massive multimodal models.

In the following table a selection of different input/output modalities.

"Interaction with prompt-commanded AI is a different from other ways of interaction with machines" [01]. It has three important properties:

  • flexibility: using of text, code, images etc.;
  • generality: applicable to broad range of tasks;
  • originality: generate original content.

"Cognitive tools are external artifacts that are used to aid the psychological capacities of the human brain in completing a cognitive task" [01] They are used to reduce the cognitive work of human brain.

Massive multimodal models are cognitive tools or extenders they can be used for simple o complex interactions. The results depend on the skill and capacity of the user in exploiting these cognitive tools.

AI IMAGE SYNTHESIS: AI Text-to-Art Generator

It is the task where AI learns to understand a description in natural language and reproduce realistic image matching the description. It combines natural language processing (NLP) and computer vision (CV). In this text-to-image tasks NLP model is the encoder and an image synthesis model as the decoder.

Our society is becoming increasingly visual. Images are a very strong means of communication and in this Artificial Intelligence is a very powerful tool.

We can create incredible images (AI Text-to-Art Generator) using AI. Here are some websites where we can create image using AI:

  • www.bing.com/images/create: AI-powered Bing using its new feature Image Creator: “Powered by the very latest DALL∙E models from our partners at OpenAI, Bing Image Creator allows you to create an image simply by using your own words to describe the picture you want to see. Now users off the waitlist can generate both written and visual content in one place from within chat.”;
  • www.midjourney.com: you can access by discord.com account.
  • openai.com/dall-e-2: DALL·E 2 is an AI system that can create realistic images and art from a description in natural language. It is not free by default.
  • YOUIMAGINE from you.com: to magically transform your ideas into stunning visuals and one-of-a-kind graphics;
  • stable-diffusion-art.com: Stable Diffusion Art:
  • www.canva.com: MagicStudio By Canva allows to supercharge your work and designs with all power of AI.
  • www.imagine.art : "Create awe-inspiring masterpieces effortlessly and explore the endless possibilities of AI generated art". 
  • davinci.ai : Create AI art using only your words in just a few seconds!

AI-powered Bing’s Image Creator

More specific is the prompt better you obtain the image that you have in mind. Bing’s Image Creator recommends you format your prompts:

Adjective + Noun + Verb + Style.

Small fox running in the forest, digital art

Simple prompt description:

The centurion in the time of the Roman Empire: the backbone of the Roman army.

in Bing's Image Creator produce in output:

Complex prompt description

"[...] the centurions must be, not so much men who are bold and contemptuous of danger, as men who are able to command, tenacious and calm, who, moreover, do not move to attack when the situation is uncertain, nor throw themselves into the heat of battle, but on the contrary know how to resist even when pressed and defeated, and are ready to die on the battlefield." POLYBIUS, HISTORIES, VI, 24, 9

Produce:

MIDJOURNEY

Let's have a look to midjourney: an AI image generator prompt. A prompt is an input that guides a computer’s AI system in producing an art.

Prompts can range from a simple text description:

The centurion in the time of the Roman Empire: the backbone of the Roman army.

that produce:

to more complicated description that involve multiple parts coming together:

"[...] the centurions must be, not so much men who are bold and contemptuous of danger, as men who are able to command, tenacious and calm, who, moreover, do not move to attack when the situation is uncertain, nor throw themselves into the heat of battle, but on the contrary know how to resist even when pressed and defeated, and are ready to die on the battlefield." POLYBIUS, HISTORIES, VI, 24, 9

Produce:

In midjourney the syntax of the prompt in order to generate an image is the following:

/imagine < description text of the image >

What you put in the prompt is very important in order to define the picture that you would have be generated by midjourney.com.

You can ask :

a photo of ...txt...

a painting of ...txt...

you can decide the subject of the photo or painting:

  • animal;
  • person;
  • landscape;
  • object;
  • and so on

You can define which details you would like to add:

1) special environment: for example on a boat, in the forest,...

2) special lighting:

  • soft lighting,
  • ring lighting,
  • neon,
  • and son on

3) colour scheme;

4) point of view:

  • camera behind;
  • camera in the front of;
  • camera beside;

5) background:

  • solid colour;
  • a nebula;
  • a forest;
  • and so on.

6) atmosphere:

  • vibrant;
  • dark;
  • and so on.

You can add more information, for example the time of the day and so on.

As an image is 1000 words you can ask midjourney.com to generate an image by an uploaded image.

You can merge multiple images into one by using the blending process.

/blend <image1> <image2> ...

You can add additional text to enrich or modify the image.

You can ask also midjourney to get the prompt back (image captioning) using this command syntax:

/describe <image>

This means please describe this image for me.

You can use a negative prompting about you don't really want in the results, putting at the end of your prompt:

--no fog or dust

The you can use other commands:

--aspect 5:4 for aspect ratio

--ar 5:4

--chaos 100 or 90

You can stop the process at a determined percent in order to have an image at different stages.

--stop 50

Human-Computer Interaction but Human-Centred

Massive multimodal models are cognitive extenders and are distinct from autonomous AI systems because they are highly user-dependent.

[01] Wout Schellaert et al., Your Prompt is My Command, Journal of Artificial Intelligence Research, 2023;

[02] Ronald T. Kneusel, How AI works, 2024;