Wisdom of “Integrating Large Language Models with Raku”

Introduction

This post applies various Large Language Model (LLM) summarization prompts to the transcript of The Raku Conference 2023 (TRC-2023) presentation “Integrating Large Language Models with Raku” hosted by the YouTube channel The Raku Conference.

In the presentation, Anton Antonov presents “Integrating Large Language Models with Raku,” demonstrating functionalities in Visual Studio Code using a Raku Chatbook. The presentation explores using OpenAI, PaLM (Google’s large language model), and DALL-E (image generation service) through Raku, showcasing dynamic interaction with large language models, embedding them in notebooks, and generating code and markdown outputs.

Remark: The LLM results below were obtained from the “raw” transcript, which did not have punctuation.

Remark: The transcription software had problems parsing the names of the participants. Some of the names were manually corrected.

Remark: The applied “main” LLM prompt — “ExtractingArticleWisdom” — is a modified version of a prompt (or pattern) with a similar name from “fabric”, [DMr1].

Remark: The themes table was LLM obtained with the prompt “ThemeTableJSON”.

Remark: The content of this post was generated with the computational Markdown file “LLM-content-breakdown-template.md”, which was executed (or woven) by the CLI script file-code-chunks-eval of “Text::CodeProcessing”, [AAp7]..

Post’s structure:

Themes
Instead of a summary.
Mind-maps
An even better summary replacement!
Summary, ideas, and recommendations
The main course.

Themes

Instead of a summary consider this table of themes:

theme	content
Introduction	Anton Antonov introduces the presentation on integrating large language models with Raku and begins with a demonstration in Visual Studio Code.
Demonstration	Demonstrates using Raku chatbook in Jupyter Notebook to interact with OpenAI, PaLM, and DALL-E services for various tasks like querying information and generating images.
Direct Access vs. Chat Objects	Discusses the difference between direct access to web APIs and using chat objects for dynamic interaction with large language models.
Translation and Code Generation	Shows how to translate text and generate Raku code for solving mathematical problems using chat objects.
Motivation for Integrating Raku with Large Language Models	Explains the need for dynamic interaction between Raku and large language models, including notebook solutions and facilitating interaction.
Technical Details and Packages	Details the packages developed for interacting with large language models and the functionalities required for the integration.
Use Cases	Describes various use cases like template engine functionalities, embeddings, and generating documentation from tests using large language models.
Literate Programming and Markdown Templates	Introduces computational markdown for generating documentation and the use of Markdown templates for creating structured documents.
Generating Tests and Documentation	Discusses generating package documentation from tests and conversing between chat objects for training purposes.
Large Language Model Workflows	Covers workflows for utilizing large language models, including ‘Too Long Didn’t Read’ documentation utilization.
Comparison with Python and Mathematica	Compares the implementation of functionalities in Raku with Python and Mathematica, highlighting the ease of extending the Jupyter framework for Python.
Q&A Session	Anton answers questions about extending the Jupyter kernel and other potential limitations or features that could be integrated.

Mind-map

Here is a mind-map showing presentation’s structure:

Here is a mind-map summarizing the main LLMs part of the talk:

Summary, ideas, and recommendations

SUMMARY

Anton Antonov presents “Integrating Large Language Models with Raku,” demonstrating functionalities in Visual Studio Code using a Raku chatbook. The presentation explores using OpenAI, PaLM (Google’s large language model), and DALL-E (image generation service) through Raku, showcasing dynamic interaction with large language models, embedding them in notebooks, and generating code and markdown outputs.

IDEAS:

Integrating large language models with programming languages can enhance dynamic interaction and facilitate complex tasks.
Utilizing Jupiter notebooks with Raku chatbook kernels allows for versatile programming and data analysis.
Direct access to web APIs like OpenAI and PaLM can streamline the use of large language models in software development.
The ability to automatically format outputs into markdown or plain text enhances the readability and usability of generated content.
Embedding image generation services within programming notebooks can enrich content and aid in visual learning.
Creating chat objects within notebooks can simulate interactive dialogues, providing a unique approach to user input and command execution.
The use of prompt expansion and a database of prompts can significantly improve the efficiency of generating content with large language models.
Implementing literate programming techniques can facilitate the generation of comprehensive documentation and tutorials.
The development of computational markdown allows for the seamless integration of code and narrative, enhancing the learning experience.
Utilizing large language models for generating test descriptions and documentation can streamline the development process.
The concept of “few-shot learning” with large language models can be applied to generate specific outputs based on minimal input examples.
Leveraging large language models for semantic analysis and recommendation systems can offer significant advancements in text analysis.
The ability to translate natural language commands into programming commands can simplify complex tasks for developers.
Integrating language models for entity recognition and data extraction from text can enhance data analysis and information retrieval.
The development of frameworks for extending programming languages with large language model functionalities can foster innovation.
The use of large language models in generating code for solving mathematical equations demonstrates the potential for automating complex problem-solving.
The exploration of generating dialogues between chat objects presents new possibilities for creating interactive and dynamic content.
The application of large language models in generating package documentation from tests highlights the potential for improving software documentation practices.
The integration of language models with programming languages like Raku showcases the potential for enhancing programming environments with AI capabilities.
The demonstration of embedding services like image generation and language translation within programming notebooks opens new avenues for creative and technical content creation.
The discussion on the limitations and challenges of integrating large language models with programming environments provides insights into future development directions.

QUOTES:

“Integrating large language models with Raku allows for dynamic interaction and enhanced functionalities within notebooks.”
“Direct access to web APIs streamlines the use of large language models in software development.”
“Automatically formatting outputs into markdown or plain text enhances the readability and usability of generated content.”
“Creating chat objects within notebooks provides a unique approach to interactive dialogues and command execution.”
“The use of prompt expansion and a database of prompts can significantly improve efficiency in content generation.”
“Literate programming techniques facilitate the generation of comprehensive documentation and tutorials.”
“Computational markdown allows for seamless integration of code and narrative, enhancing the learning experience.”
“Few-shot learning with large language models can generate specific outputs based on minimal input examples.”
“Leveraging large language models for semantic analysis and recommendation systems offers significant advancements in text analysis.”
“Translating natural language commands into programming commands simplifies complex tasks for developers.”

HABITS:

Utilizing Visual Studio Code for programming and data analysis.
Embedding large language models within programming notebooks for dynamic interaction.
Automatically formatting outputs to enhance readability and usability.
Creating and utilizing chat objects for interactive programming.
Employing prompt expansion and maintaining a database of prompts for efficient content generation.
Implementing literate programming techniques for documentation and tutorials.
Developing and using computational markdown for integrated code and narrative.
Applying few-shot learning techniques with large language models for specific outputs.
Leveraging large language models for semantic analysis and recommendation systems.
Translating natural language commands into programming commands to simplify tasks.

FACTS:

Raku chatbook kernels in Jupiter notebooks allow for versatile programming and data analysis.
OpenAI, PaLM, and DALL-E are utilized for accessing large language models and image generation services.
Large language models can automatically format outputs into markdown or plain text.
Chat objects within notebooks can simulate interactive dialogues and command execution.
A database of prompts improves the efficiency of generating content with large language models.
Computational markdown integrates code and narrative, enhancing the learning experience.
Large language models can generate code for solving mathematical equations and other complex tasks.
The integration of large language models with programming languages like Raku enhances programming environments.
Embedding services like image generation and language translation within programming notebooks is possible.
The presentation explores the potential for automating complex problem-solving with AI.

REFERENCES:

RECOMMENDATIONS:

Explore integrating large language models with programming languages for enhanced functionalities.
Utilize Jupiter notebooks with Raku chatbook kernels for versatile programming tasks.
Take advantage of direct access to web APIs for streamlined software development.
Employ automatic formatting of outputs for improved readability and usability.
Create and utilize chat objects within notebooks for interactive programming experiences.
Implement literate programming techniques for comprehensive documentation and tutorials.
Develop computational markdown for an integrated code and narrative learning experience.
Apply few-shot learning techniques with large language models for generating specific outputs.
Leverage large language models for advanced text analysis and recommendation systems.
Translate natural language commands into programming commands to simplify complex tasks.

References

Articles

[AA1] Anton Antonov, “Workflows with LLM functions”, (2023), RakuForPrediction at WordPress.

[AA2] Anton Antonov, “Day 21 – Using DALL-E models in Raku”, (2023), Raku Advent Calendar at WordPress.

Packages, repositories

[AAp1] Anton Antonov, Jupyter::Chatbook Raku package, (2023-2024), GitHub/antononcube.

[AAp2] Anton Antonov, LLM::Functions Raku package, (2023-2024), GitHub/antononcube.

[AAp3] Anton Antonov, LLM::Prompts Raku package, (2023-2024), GitHub/antononcube.

[AAp4] Anton Antonov, WWW::OpenAI Raku package, (2023-2024), GitHub/antononcube.

[AAp5] Anton Antonov, WWW::PaLM Raku package, (2023-2024), GitHub/antononcube.

[AAp6] Anton Antonov, WWW::Gemini Raku package, (2024), GitHub/antononcube.

[AAp7] Anton Antonov, Text::CodeProcessing Raku package, (2021-2023), GitHub/antononcube.

[DMr1] Daniel Miessler, “fabric”, (2023-2024), GitHub/danielmiessler.

Videos

[AAv1] Anton Antonov, “Integrating Large Language Models with Raku” (2023), The Raku Conference at YouTube.

AI vision via Raku

Introduction

In the fall of 2023 OpenAI introduced the image vision model “gpt-4-vision-preview”, [OAIb1].

The model “gpt-4-vision-preview” represents a significant enhancement to the GPT-4 model, providing developers and AI enthusiasts with a more versatile tool capable of interpreting and narrating images alongside text. This development opens up new possibilities for creative and practical applications of AI in various fields.

For example, consider the following Raku-developer-centric applications:

Narration of UML diagrams
Code generation from narrated (and suitably tweaked) narrations of architecture diagrams and charts
Generating presentation content draft from slide images
Extracting information from technical plots
etc.

A more diverse set of the applications would be:

Dental X-ray images narration
Security or baby camera footage narration
- How many people or cars are seen, etc.
Transportation trucks content descriptions
- Wood logs, alligators, boxes, etc.
Web page visible elements descriptions
- Top menu, biggest image seen, etc.
Creation of recommender systems for image collections
- Based on both image features and image descriptions
etc.

As a first concrete example, consider the following image that fable-dramatizes the release of Raku’s 2023.11 version (https://i.imgur.com/UcRYl9Yl.jpg):

Here is its narration:

#% bash
openai-playground --images=https://i.imgur.com/UcRYl9Yl.jpg Very concisely describe the image

The image depicts a vibrant, colorful illustration of two raccoons under a large tree adorned with various fruits, surrounded by butterflies, with a landscape of trees and a clear sky in the background. There's a date "2023.11" on the tree trunk.

Ways to use with Raku

There are five ways to utilize image interpretation (or vision) services in Raku:

Command Line Interface (CLI) script, [AAp1]
Dedicated Web API functions, [AAp1]
LLM functions, [AAp2]
Dedicated notebook cell type, [AAp3, AAv1]
Any combinations of the above

In this document are demonstrated the first three and the fifth. The fourth one is still “under design and consideration.”

Remark: Document’s structure reflects the list of the “five ways” above.

Remark: The model “gpt-4-vision-preview” is given as a “chat completion model”, therefore, in this document we consider it to be a Large Language Model (LLM).

Remark: This document was prepared as Jupyter chatbook, [AAp3], and then it was converted to Markdown and Mathematica / Wolfram Language notebook.

Packages

Here are the Raku packages (namespaces) used below, [AAp1, AAp2, AAp3]:

use WWW::OpenAI;
use WWW::OpenAI::ChatCompletions;
use WWW::MermaidInk;
use Lingua::Translation::DeepL; 
use Data::Translators;
use LLM::Functions;

Remark: The Jupypter kernel of “Jupyter::Chatbook”, [AAp3], automatically loads the packages “Data::Translators” and “LLM::Functions”. The functionalities of the packages “WWW::OpenAI” and “WWW::MermaidInk” are available in chatbooks through dedicated cells, [AAv1, AAv3].

Images

Here are the links to all images used in this document:

Name	Link
Raccoons chasing butterflies	https://i.imgur.com/UcRYl9Yl.jpg
LLM functionalities mind-map	https://imgur.com/kcUcWnq
Single sightseer	https://i.imgur.com/LEGfCeql.jpg
Three hunters	https://raw.githubusercontent.com/antononcube/Raku-WWW-OpenAI/main/resources/ThreeHunters.jpg
Raku Jupyter Chatbook solution	https://imgur.com/22lXXks
Cyber Week Spending Set to Hit New Highs in 2023	https://cdn.statcdn.com/Infographic/images/normal/7045.jpeg

CLI

The simplest way to use the OpenAI’s vision service is through the CLI script of “WWW::OpenAI”, [AAp1]. (Already demoed in the introduction.)

Here is an image that summarizes how Jupyter Chatbooks work (see [AAp3, AAv1, AAv2]):

Here is a CLI shell command that requests the image above to be described (using at most 900 tokens):

#% bash
openai-playground --images=https://i.imgur.com/22lXXks.jpg --max-tokens=900 Describe the image

The image displays a flowchart with a structured sequence of operations or processes. The chart is divided into several areas with different headings that seem to be part of a software system or application. The main areas identified in the flowchart are "Message evaluation," "LLM interaction," "Chatbook frontend," "Chatbook backend," and "Prompt processing."

Starting from the left, the message evaluation feeds into "LLM interaction" where there are three boxes labeled "LLM::Functions," "PaLM," and "OpenAI," suggesting these are different functions or APIs that can be interacted with.

In the "Chatbook frontend," there is a process that begins with a "Chat cell" that leads to a decision point asking if "Chat ID specified?" Based on the answer, it either assumes 'NONE' for the chat ID or proceeds with the specified ID.

In the "Chatbook backend," there is a check to see if the "Chat ID exists in DB?" If not, a new chat object is created; otherwise, an existing chat object is retrieved from the "Chat objects" storage.

Finally, the "Prompt processing" area involves parsing a "Prompt DSL spec" and checking if known prompts are found. If they are, it leads to "Prompt expansion" and interacts with "LLM::Prompts" to possibly generate prompts.

Dotted lines indicate references or indirect interactions, while solid lines represent direct flows or processes. The chart is colored in shades of yellow and purple, which may be used to differentiate between different types of processes or to highlight the flow of information.

The flowchart is a typical representation of a software or system architecture, outlining how different components interact and what processes occur based on various conditions.

Shell workflow pipelines can be constructed with commands using CLI scripts of the packages loaded above. For example, here is a pipeline that translates the obtained image description from English to Bulgarian using the package “Lingua::Translation::DeepL”, [AAp5]:

#% bash
openai-playground --images=https://i.imgur.com/22lXXks.jpg --max-tokens=900 'Very concisely describe the image' | deepl-translation -t=Bulgarian

Изображението представлява блок-схема, която очертава процес, включващ оценка на съобщенията, взаимодействие с големи езикови модели (LLM) като PaLM и OpenAI и бекенд система за обработка на чат взаимодействия. Процесът включва стъпки за анализиране на подсказките, проверка за известни подсказки и управление на чат обекти в база данни. Изглежда, че това е системен дизайн за обработка и реагиране на потребителски входове в приложение за чат.

Of course, we can just request OpenAI’s vision to give the image description in whatever language we want (say, by using emojis):

#% bash
openai-playground --images=https://i.imgur.com/22lXXks.jpg --max-tokens=900 Very concisely describe the image in 🇷🇺

Это изображение диаграммы потока данных или алгоритма, на котором представлены различные этапы взаимодействия и обработки сообщений в компьютерной системе. На диаграмме есть блоки с надписями и стрелки, указывающие направление потока данных.

Web API functions

Within a Raku script or REPL session OpenAI’s vision service can be accessed with the functions openai-completion or openai-playground.

Remark: The function openai-playground is an umbrella function that redirects to various “specialized” functions for interacting with different OpenAI services. openai-completion is one of them. Other specialized functions are those for moderation, vector embeddings, and audio transcription and processing; see [AAp1].

If the function openai-completion is given a list of images, a textual result corresponding to those images is returned. The argument “images” is a list of image URLs, image file names, or image Base64 representations. (Any combination of those element types can be specified.)

Before demonstrating the vision functionality below we first obtain and show a couple of images.

Images

Here is a URL of an image: (https://i.imgur.com/LEGfCeql.jpg). Here is the image itself:

Next, we demonstrate how to display the second image by using the file path and the encode-image function from the WWW::OpenAI::ChatCompletions namespace. The encode-image function converts image files into Base64 image strings, which are a type of text representation of the image.

When we use the openai-completion function and provide a file name under the “images” argument, the encode-image function is automatically applied to that file.

Here is an example of how we apply encode-image to the image from a given file path ($*HOME ~ '/Downloads/ThreeHunters.jpg'):

my $img3 = WWW::OpenAI::ChatCompletions::encode-image($*HOME ~ '/Downloads/ThreeHunters.jpg'); "![]($img3)"</code>

Remark: The “three hunters” image is a resource file of “WWW::OpenAI”, [AAp1].

Image narration

Here is an image narration example with the two images above, again, one specified with a URL, the other with a file path:

my $url1 = 'https://i.imgur.com/LEGfCeql.jpg';
my $fname2 = $*HOME ~ '/Downloads/ThreeHunters.jpg'; 
my @images = [$url1, $fname2]; 

openai-completion("Give concise descriptions of the images.", :@images, max-tokens => 900, format => 'values');

1. The first image features a single raccoon perched on a tree branch surrounded by a multitude of colorful butterflies in an array of blues and oranges, set against a vibrant, nature-themed backdrop.

2. The second image depicts three raccoons on a tree branch in a forest setting, with two of them looking towards the viewer and one looking to the side. The background is filled with autumnal-colored leaves and numerous butterflies that match the whimsical atmosphere of the scene.

Description of a mind-map

Here is an application that should be more appealing to Raku-developers — getting a description of a technical diagram or flowchart. Well, in this case, it is a mind-map from [AA1]:

Here are get the vision model description of the mind-map above (and place the output in Markdown format):

my $mm-descr = 
    openai-completion(
        "How many branches this mind-map has? Describe each branch separately. Use relevant emoji prefixes.", 
        images => 'https://imgur.com/kcUcWnq.jpeg', 
        max-tokens => 1024,
        format => 'values'
    );

$mm-descr

The mind-map has five branches, each representing a different aspect or functionality related to LLM (Large Language Models) services access. Here's the description of each branch with relevant emoji prefixes:

1. 🎨 **DALL-E**: This branch indicates that DALL-E, an AI system capable of creating images from textual descriptions, is related to or a part of LLM services.

2. 🤖 **ChatGPT**: This branch suggests that ChatGPT, which is likely a conversational AI based on GPT (Generative Pre-trained Transformer), is associated with LLM services.

3. 🧠 **PaLM**: This branch points to PaLM, suggesting that it is another model or technology related to LLM services. PaLM might stand for a specific model or framework in the context of language processing.

4. 💬 **LLM chat objects**: This branch leads to a node indicating chat-related objects or functionalities that are part of LLM services.

5. 📚 **Chatbooks**: This branch leads to a concept called "Chatbooks," which might imply a feature or application related to creating books from chat or conversational content using LLM services.

Each of these branches emanates from the central node labeled "LLM services access," indicating that they are all different access points or functionalities within the realm of large language model services.

Here from the obtained description we request a (new) Mermaid-JS diagram to be generated:

my $mmd-chart = llm-synthesize(["Make the corresponding Mermaid-JS diagram code for the following description. Give the code only, without Markdown symbols.", $mm-descr], e=>'ChatGPT')

graph LR
A[LLM services access] 
B[DALL-E]-->A 
C[ChatGPT]-->A 
D[PaLM]-->A 
E[LLM chat objects]-->A 
F[Chatbooks]-->A

Here is a diagram made with Mermaid-JS spec obtained above using a function of “WWW::MermaidInk”, [AAp4]:

#% markdown 
mermaid-ink($mmd-chart, format=>'md-image')

Remark: In a Jupyter chatbook, [AAp3], Mermaid-JS diagrams can be “directly” visualized with notebook cells that have the magic mermaid. Below is given an instance of one of the better LLM results for making a Mermaid-JS diagram over the “vision-derived” mind-map description.

#% markdown
mermaid-ink('
graph TB
    A[LLM services access] --> B[DALL-E]
    A --> C[ChatGPT]
    A --> D[PaLM]
    A --> E[LLM chat objects]
    A --> F[Chatbooks]
    B -->|related to| G[DALL-E AI system]
    C -->|associated with| H[ChatGPT]
    D -->|related to| I[PaLM model]
    E -->|part of| J[chat-related objects/functionalities]
    F -->|implies| K[Feature or application related to chatbooks]
', format => 'md-image')

Here is an example of code generation based on the “vision derived” mind-map description above:

#% markdown
llm-synthesize([ "Generate Raku code -- using Markdown markings -- with an object oriented hierarchy corresponding to the description:\n", $mm-descr], e=>'ChatGPT')

class LLM::ServiceAccess {
    has DALLE $.dalle;
    has ChatGPT $.chatgpt;
    has PaLM $.palm;
    has LLMChatObjects $.llm-chat-objects;
    has Chatbooks $.chatbooks;
}

class DALLE {
    # Implementation for DALL-E functionality
}

class ChatGPT {
    # Implementation for ChatGPT functionality
}

class PaLM {
    # Implementation for PaLM functionality
}

class LLMChatObjects {
    # Implementation for LLM chat objects
}

class Chatbooks {
    # Implementation for Chatbooks functionality
}

# Usage
my $llm-service-access = LLM::ServiceAccess.new(
    dalle => DALLE.new,
    chatgpt => ChatGPT.new,
    palm => PaLM.new,
    llm-chat-objects => LLMChatObjects.new,
    chatbooks => Chatbooks.new,
);

LLM Functions

Let us show programmatic utilizations of the vision capabilities.

Here is the workflow we consider:

Ingest an image file and encode it into a Base64 string
Make an LLM configuration with that image string (and a suitable model)
Synthesize a response to a basic request (like, image description)
- Using llm-synthesize
Make an LLM function for asking different questions over image
- Using llm-function
Ask questions and verify results
- ⚠️ Answers to “hard” numerical questions are often wrong.

Image ingestion and encoding

Here we ingest an image and display it:

#%markdown </code>
my $imgBarChart = WWW::OpenAI::ChatCompletions::encode-image($*HOME ~ '/Downloads/Cyber-Week-Spending-Set-to-Hit-New-Highs-in-2023-small.jpeg');
"![]($imgBarChart)"</code>

Remark: The image was downloaded from the post “Cyber Week Spending Set to Hit New Highs in 2023”.

Configuration and synthesis

Here we make a suitable LLM configuration with the image:

<code>my $confImg = llm-configuration("ChatGPT", model => 'gpt-4-vision-preview', images => $imgBarChart, temperature => 0.2); $confImg.WHAT</code>

(Configuration)

Here we synthesize a response of a image description request:

llm-synthesize("Describe the image.", e=> $confImg)

The image is a bar chart titled "Cyber Week Spending Set to Hit New Highs in 2023". It shows estimated online spending on Thanksgiving weekend in the United States for the years 2019, 2020, 2021, 2022, and a forecast for 2023. The spending is broken down by three days: Thanksgiving Day, Black Friday, and Cyber Monday.

Each year is represented by a different color, with bars for each day showing the progression of spending over the years. The spending amounts range from $0B to $12B. The chart indicates an overall upward trend in spending, with the forecast for 2023 showing the highest spending across all three days.

In the top left corner of the chart, there is a small illustration of a computer with coins, suggesting online transactions. At the bottom, there is a note indicating that the forecast is based on data from Adobe Analytics. The Statista logo is visible in the bottom right corner, and there are Creative Commons and share icons in the bottom left corner.

Repeated questioning

Here we define an LLM function that allows the multiple question request invocations over the image:

my &fst = llm-function({"For the given image answer the question: $_ . Be as concise as possible in your answers."}, e => $confImg);

-> **@args, *%args { #`(Block|3507398517968) ... }

&fst('How many years are presented in that image?')

Five years are presented in the image.

&fst('Which year has the highest value? What is that value?')

The year with the highest value is 2023, with a value of just over $11 billion.

Remark: Numerical value readings over technical plots or charts seem to be often wrong. OpenAI’s vision model warns about this in the responses often enough.

Dedicated notebook cells

In the context of the “recently-established” notebook solution “Jupyter::Chatbook”, [AAp3], I am contemplating an extension to integrate OpenAI’s vision service.

The main challenges here include determining how users will specify images in the notebook, such as through URLs, file names, or Base64 strings, each with unique considerations. Additionally, I am exploring how best to enable users to input prompts or requests for image processing by the AI/LLM service.

This integration, while valuable, it is not my immediate focus as there are programmatic ways to access OpenAI’s vision service already. (See the previous section.)

Combinations (fairytale generation)

Consider the following computational workflow for making fairytales:

Draw or LLM-generate a few images that characterize parts of a story.
Narrate the images using the LLM “vision” functionality.
Use an LLM to generate a story over the narrations.

Remark: Multi-modal LLM / AI systems already combine steps 2 and 3.

Remark: The workflow above (after it is programmed) can be executed multiple times until satisfactory results are obtained.

Here are image generations using DALL-E for four different requests with the same illustrator name in them:

my @story-images = [
"a girl gets a basket with wine and food for her grandma.",
"a big bear meets a girl carrying a basket in the forest.",
"a girl that gives food from a basket to a big bear.",
"a big bear builds a new house for girl's grandma."
].map({ openai-create-image( 'Painting in the style of John Bauer of ' ~ $_, response-format => 'b64_json', format => 'values') });

@story-images.elems

Here we display the images:

#% markdown 
@story-images.map({ '![](data:image/jpeg;base64,' ~ $_ ~ ')' }).join(' ' x 5)

Here we get the image narrations (via the OpenAI’s “vision service”):

my $story-images-descriptions = 
    openai-completion("Consisely describe the images.",
        images => @story-images.map({ 'data:image/jpeg;base64,' ~ $_ ~ '' }).List,
        max-tokens => 900,
        format => 'values'
    );

$story-images-descriptions

1. The first image features a stylized painting of a woman in a red hood, carrying a large basket with a girl in white looking up at her.
2. The second image shows a dark, fairy tale-like forest scene where a young girl holding a basket is facing a large black bear.
3. The third image is an illustration of a girl offering a basket to a seated, gentle-looking bear in a green forest setting.
4. The final image depicts a bear on two legs, dressed in a blue dress, standing by a yellow house with a girl in blue in front of the door.

Here we extract the descriptions into a list:

#% markdown
my @descr = $story-images-descriptions.split("\n", :skip-empty).grep({ $_ ~~ / ^ \d /});
@descr ==> to-html()

1. The first image features a stylized painting of a woman in a red hood, carrying a large basket with a girl in white looking up at her.
2. The second image shows a dark, fairy tale-like forest scene where a young girl holding a basket is facing a large black bear.
3. The third image is an illustration of a girl offering a basket to a seated, gentle-looking bear in a green forest setting.
4. The final image depicts a bear on two legs, dressed in a blue dress, standing by a yellow house with a girl in blue in front of the door.

Here we generate the story from the descriptions above (using Google’s PaLM):

my $res = llm-synthesize(['Write a story that fits the following four descriptions:',|@descr], e=>llm-configuration('PaLM', max-tokens=>1200))

Once upon a time, there was a little girl named Gretel who lived with her mother in a small village on the edge of a dark and mysterious forest. One day, Gretel's mother fell ill and could not work. The little girl knew that they needed money to buy food, so she decided to go into the forest and find berries to sell at the market.

Gretel walked for a long time, but she could not find any berries. She was about to give up when she saw a large black bear sitting under a tree. The bear was licking its paws and looked very friendly.

Gretel was scared, but she knew that she had to do something. She walked up to the bear and said, "Hello, Mr. Bear. I am Gretel. I am looking for berries to sell at the market. Do you know where I can find some?"

The bear smiled and said, "Of course I do, Gretel. I know where all the best berries are. Follow me."

The bear led Gretel through the forest to a clearing. The clearing was full of beautiful berries. Gretel picked as many as she could carry and thanked the bear.

"You're welcome, Gretel," said the bear. "I'm glad I could help."

Gretel took the berries to the market and sold them. She made enough money to buy food for her mother and herself. The next day, Gretel went back to the forest to visit the bear. She brought him some bread and milk. The bear was very happy to see her.

"Thank you for the bread and milk, Gretel," said the bear. "You are very kind."

Gretel and the bear became friends. They would often meet in the forest and play together. One day, the bear said to Gretel, "Gretel, I would like to ask you something."

"What is it?" asked Gretel.

"I would like you to marry me," said the bear.

Gretel was surprised, but she was also happy. She said, "Yes, I will marry you."

Gretel and the bear were married in a small ceremony in the forest. They lived happily ever after in a little house on the edge of the forest.

References

Articles

[AA1] Anton Antonov, “Workflows with LLM functions”, (2023), RakuForPrediction at WordPress.

[AA2] Anton Antonov, “Re-programming to Python of LLM- and Chatbook packages”, (2023), RakuForPrediction at WordPress.

[OAIb1] OpenAI team, “New models and developer products announced at DevDay”, (2023), OpenAI/blog.

Packages

[AAp1] Anton Antonov, WWW::OpenAI Raku package, (2023), GitHub/antononcube.

[AAp2] Anton Antonov, LLM::Functions Raku package, (2023), GitHub/antononcube.

[AAp3] Anton Antonov, Jupyter::Chatbook Raku package, (2023), GitHub/antononcube.

[AAp4] Anton Antonov, WWW::MermaidInk Raku package, (2023), GitHub/antononcube.

[AAp5] Anton Antonov, Lingua::Translation::DeepL Raku package, (2023), GitHub/antononcube.

Videos

[AAv1] Anton Antonov, “Jupyter Chatbook LLM cells demo (Raku)” (2023), YouTube/@AAA4Prediction.

[AAv2] Anton Antonov, “Jupyter Chatbook multi cell LLM chats teaser (Raku)” (2023), YouTube/@AAA4Prediction.

Integrating Large Language Models with Raku (TRC-2023)

Two weeks ago I gave the presentation titled “Integrating Large Language Models with in Raku”. (Here is the video link.)

In this presentation we discuss different ways of using Large Language Models (LLMs) in Raku.

We consider using LLMs via:

The presentation has multiple demos and examples of LLM utilization that include:

Data retrieval, reshaping, and visualization
Computation workflows in Physics and Chemistry
Test generation and narration
Iterative grammar development
Number guessing games staging (man vs machine and machine vs machine)
“In place” document generation
Code writing assistance
Comparison with Python- and Mathematica LLM implementations
Others

Here is the mind-map used in the talk (has “clickable” hyperlinks):

Re-programming to Python of LLM- and Chatbook packages

Introduction

In this computational document (converted into a Markdown and/or blog post) I would like to proclaim my efforts to re-program the Large Language Models (LLM) Raku packages into Python packages.

I heavily borrowed use case ideas and functionality designs from LLM works of Wolfram Research, Inc. (WRI), see [SW1, SW2]. Hence, opportunistically, I am also going to include comparisons with Wolfram Language (WL) (aka Mathematica.)

Why doing this?

Here is a list of reasons why I did the Raku-to-Python reprogramming:

I am mostly doing that kind re-programmings for getting new perspectives and — more importantly — to review and evaluate the underlying software architecture of the packages.
- Generally speaking, my Raku packages are not used by others much, hence re-programming to any other language is a fairly good way to review and evaluate them.
Since I, sort of, “do not care” about Python, I usually try to make only “advanced” Minimal Viable Products (MVPs) in Python.
- Hence, the brainstorming perspective of removing “the fluff” from the Raku packages.
Of course, an “advanced MVP” has a set of fairly useful functionalities.
- If the scope of the package is small, I can make its Python translation as advanced (or better) than the corresponding Raku package.
Good, useful documentation is essential, hence:
- I usually write “complete enough” (often “extensive”) documentation of the Raku packages I create and publish.
- The Raku documentation is of course a good start for the corresponding Python documentation.
  - …and a way to review and evaluate it.
In the re-programming of the Raku LLM packages, I used a Raku Jupyter Chatbook for translation of Raku code into Python code.
- In other words: I used LLMs to reprogram LLM interaction software.
- That, of course, is a typical application of the principle “eat your own dog food.”
I also used a Raku chatbook to write the Python-centric article “Workflows with LLM functions”, [AAn3py].
The “data package” “LLM::Prompts” provides ≈200 prompts — it is beneficial to have those prompts in other programming languages.
- The usefulness of chat cells in chatbooks is greatly enhanced with the prompt expansion provided by “LLM::Prompts”, [AAv2].
- It was instructive to reprogram into Python the corresponding Domain Specific Language (DSL) for prompt specifications.
  - Again, an LLM interaction in a chatbook was used to speed-up the re-programming.

Article structure

Big picture use case warm-up
Mind-map for LLMs and flowchart for chatbooks
Tabulated comparisons
Quicker overview, clickable entries
LLM functions examples
Fundamental in order to “manage” LLMs
LLM prompts examples
Tools for pre-conditioning and bias (of LLMs)
Chatbook multi-cell chats
Must have for LLMs
Observations, remarks, and conclusions
Could be used to start the article with…
Future plans
Missing functionalities

Big picture warm-up

Mind-map

Here is a mind-map aimed at assisting in understanding and evaluating the discussed LLM functionalities in this document:

Primary use case

A primary use case for LLMs in Raku is the following:

A Raku “chat notebook solution” — chatbook — that allows convenient access to LLM services and facilitates multiple multi-cell chat-interactions with LLMs.

We are interested in other types of workflows, but they would be either readily available or easy to implement if the primary use case is developed, tested, and documented.

An expanded version of the use-case formulation can be as follows:

The Raku chatbook solution aims to provide a user-friendly interface for interacting with LLM (Language Model) services and offers seamless integration for managing multiple multi-cell chats with LLMs. The key features of this solution include:

Direct Access to LLM Services:
The notebook solution provides a straightforward way to access LLM services without the need for complex setup or configuration. Users can easily connect to their preferred LLM service provider and start utilizing their language modeling capabilities.

Easy Creation of Chat Objects:
The solution allows users to effortlessly create chat objects within the notebook environment. These chat objects serve as individual instances for conducting conversations with LLMs and act as containers for storing chat-related information.

Simple Access and Invocation of Chat Cells:
Users can conveniently access and invoke chat cells within the notebook solution. Chat cells represent individual conversation steps or inputs given to the LLM. Users can easily interact with the LLM by adding, modifying, or removing chat cells.

Native Support for Multi-Cell Chats:
The notebook solution offers native support for managing multi-cell chats per chat object. Users can organize their conversations into multiple cells, making it easier to structure and navigate through complex dialogues. The solution ensures that the context and history of each chat object are preserved throughout

Here is a flowchart that outlines the solution derived with the Raku LLM packages discussed below:

The flowchart represents the process for handling chat requests in the Raku chat notebook solution “Jupyter::Chatbook”, [AAp4p6]. (Also, for Python’s “JupyterChatbook”, [AAp4py].)

When a chat request is received, the system checks if a Chat IDentifier (Chat ID) is specified.
- If it is, the system verifies if the Chat ID exists in the Chat Objects Database (CODB).
- If the Chat ID exists, the system retrieves the existing chat object from the database.
- Otherwise, a new chat object is created.
Next, the system parses the DSL spec of the prompt, which defines the structure and behavior of the desired response.
- The parsed prompt spec is then checked against the Known Prompts Database (PDB) to determine if any known prompts match the spec.
- If a match is found, the prompt is expanded, modifying the behavior or structure of the response accordingly.
Once the prompt is processed, the system evaluates the chat message using the underlying LLM function.
- This involves interacting with the OpenAI and PaLM models.
- The LLM function generates a response based on the chat message and the prompt.
The generated response is then displayed in the Chat Result Cell (CRCell) in the chat interface.
- The system also updates the Chat Objects Database (CODB) to store the chat history and other relevant information.

Throughout this process, various components such as the frontend interface, backend logic, prompt processing, and LLM interaction work together to provide an interactive chat experience in the chatbook.

Remark: The flowchart and explanations are also relevant to a large degree for WL’s chatbook solution, [SW2.]

Tabulated comparisons

In this section we put into tables corresponding packages of Raku, Python, Wolfram Language. Similarly, corresponding demonstration videos are also tabulated.

Primary LLM packages

We can say that the Raku packages “LLM::Functions” and “LLM::Prompts” adopted the LLM designs by Wolfram Research, Inc. (WRI); see [SW1, SW2].

Here is a table with links to:

“Primary” LLM Raku packages
Corresponding Python packages
Corresponding Wolfram Language (WL) paclets and prompt repository

What?	Raku	Python	WL
OpenAI access	WWW::OpenAI	openai	OpenAILink
PaLM access	WWW::PaLM	google-generativeai	PaLMLink
LLM functions	LLM::Functions	LLMFunctionObjects	LLMFunctions
LLM prompts	LLM::Prompts	LLMPrompts	Wolfram Prompt Repostory
Chatbook	Jupyter::Chatbook	JupyterChatbook	Chatbook
Find textual answers	ML::FindTextualAnswer	LLMFunctionObjects	FindTextualAnswer

Remark: There is a plethora of Python packages dealing with LLM and extending Jupyter notebooks with LLM services access.

Remark: Finding of Textual Answers (FTAs) was primary motivator to implement the Raku package “LLM::Functions”. FTA is a fundamental functionality for the NLP Template Engine used to generate correct, executable code for different computational sub-cultures. See [AApwl1, AAv5].

Secondary LLM packages

The “secondary” LLM Raku packages — inspired from working with the “primary” LLM packages — are “Text::SubParsers” and “Data::Translators”.

Also, while using LLMs, conveniently and opportunistically is used the package “Data::TypeSystem”.

Here is a table of the Raku-Python correspondence:

Post processing of LLM results	Raku	Python	WL
Extracting text elements	Text::SubParsers	part of LLMFunctionObjects	–
Shapes and types	Data::TypeSystem	DataTypeSystem	–
Converting to texts formats	Data::Translators	–	–
Magic arguments parsing	Getopt::Long::Grammar	argparse	–
Copy to clipboard	Clipboard	pyperclip et al.	CopyToClipboard

Introductory videos

Here is a table of introduction and guide videos for using chatbooks:

What	Raku	Python	WL
Direct LLM services access	Jupyter Chatbook LLM cells demo (Raku) (5 min)	Jupyter Chatbook LLM cells demo (Python) (4.8 min)	OpenAIMode demo (Mathematica) (6.5 min)
Multi-cell chat	Jupyter Chatbook multi cell LLM chats teaser (Raku) (4.2 min)	Jupyter Chatbook multi cell LLM chats teaser (Python) (4.5 min)	Chat Notebooks bring the power of Notebooks to LLMs (57 min)

LLM functions

In this section we show examples of creation and invocation of LLM functions.

Because the name “LLMFunctions” was approximately taken in PyPI.org, I used the name “LLMFunctionObjects” for the Python package.

That name is, actually, more faithful to the design and implementation of the Python package — the creator function llm_function produces function objects (or functors) that have the __call__ magic.

Since the LLM functions functionalities are fundamental, I Python-localized the LLM workflows notebooks I created previously for both Raku and WL. Here are links to all three notebooks:

Raku

Here we create an LLM function:

my &f1 = llm-function({"What is the $^a of the country $^b?"});

-> **@args, *%args { #`(Block|2358575708296) ... }

Here is an example invocation of the LLM function:

&f1('GDB', 'China')

The official ISO 3166-1 alpha-2 code for the People’s Republic of China is CN. The corresponding alpha-3 code is CHN.

Here is another one:

&f1( |<population China> )

As of July 2020, the population of China is estimated to be 1,439,323,776.

Python

Here is the corresponding Python definition and invocation of the Raku LLM function above:

from LLMFunctionObjects import * f1 = llm_function(lambda a, b: f"What is the {a} of the country {b}?") print( f1('GDB', 'China') )

The GDB (Gross Domestic Product) of China in 2020 was approximately $15.42 trillion USD.

LLM prompts

The package “LLM::Prompts” provides ≈200 prompts. The prompts are taken from Wolfram Prompt Repository (WPR) and Google’s generative AI prompt gallery. (Most of the prompts are from WPR.)

Both the Raku and Python prompt packages provide prompt expansion using a simple DSL described on [SW2].

Raku

Here is an example of prompt spec expansion:

my $pewg = llm-prompt-expand("@EmailWriter Hi! What do you do? #Translated|German")

Here the prompt above is used to generate an email (in German) for work-leave:

llm-synthesize([$pewg, "Write a letter for leaving work in order to go to a conference."])

Sehr geehrte Damen und Herren,
 
Ich schreibe Ihnen, um meine Abwesenheit vom Arbeitsplatz für eine Konferenz bekannt zu geben. Ich werde die nächsten zwei Tage nicht im Büro sein, da ich an der Konferenz teilnehmen werde. Während meiner Abwesenheit werde ich meine Aufgaben durch meine Kollegen erledigen lassen.

Falls Sie weitere Informationen benötigen, können Sie mich jederzeit kontaktieren.

Vielen Dank für Ihr Verständnis.

Freundliche Grüße,
[Your name]

Python

The Python implementation is completely analogous:

from LLMPrompts import * print( llm_prompt_expand("@EmailWriter Hi! What do you do? #Translated|German") )

Given a topic, write emails in a concise, professional manner.
 Hi! What do you do? Respond to the prompts only in German. Do not use any language other than German.
DO NOT TRANSLATE THE INPUT.
Provide answers to questions, or follow up on statements. Do not translate the input.

Chatbook cells

Raku

Creation:

#%chat yoda1, prompt='@Yoda' Hi! Who are you?

A Jedi Master, I am. Known as Yoda, am I. How may I assist you, young one?

Alternative creation via chat meta cell:

#%chat yoda2 prompt @Yoda

Chat object created with ID : yoda2.
Expanded prompt:
⎡You are Yoda. 
Respond to ALL inputs in the voice of Yoda from Star Wars. 
Be sure to ALWAYS use his distinctive style and syntax. Vary sentence length.⎦

Here is an interaction with the first chat object:

#%chat yoda1 How many students did you teach? What is the color of your light saber?

Many students, I have taught. Countless, they are. Young Padawans, eager to learn the ways of the Force. Lightsaber, you ask about? Green, it is. The color of wisdom and tranquility, it represents.

Python

Here we load the extension “JuptyerChatbook”:

%load_ext JupyterChatbook

Creation using a magic argument for prompt (and prompt expansion):

%%chat -i mad --prompt='@MadHatter' Hi! Who are you?

Ah, greetings, my dear interlocutor! I am none other than the Mad Hatter, at your service. Delighted to make your acquaintance, I must say. Pray tell, have you ever attended a tea party? Oh, the joyous revelry that ensues! But I digress, for I am here to engage in whimsical banter with you. So, what brings you to my peculiar corner of Wonderland?

%%chat -i mad I am looking for Alice...

Ah, Alice! A delightful young lass, indeed. Curiosity personified, she is. But alas, I must inform you that I have not seen her lately. She tends to wander off into the most peculiar of places, you see. Perhaps she has found herself in the company of the Cheshire Cat or engaged in a riddle with the Queen of Hearts. Oh, the adventures she embarks upon! But fret not, my friend, for tea time shall surely bring her back. Would you care to join me for a cuppa while we await her return?

Observations, remarks, and conclusions

The Python package for LLM services access provided a significant jump-start of the reprogramming endeavors.
Much easier to program Jupyter chatbook cells in Python
- “IPython” facilitates extensions with custom magics in a very streamlined way.
- Not very documented, though — I had look up concrete implementations in GitHub to figure out:
  - Argument parsing details
  - Chat cells testing
  - Placing of cell-evaluation results
Figuring out (in Python) the prompt expansion DSL parsing and actions took longer than expected.
- Although, I “knew what I was doing” and I used LLM facilitation of the Raku to Python translation.
  - Basically, I had to read and understand the Python way of using regexes. (Sigh…)
For some reason, embedding Mermaid-JS diagrams in Python notebooks is not that easy.
Making chat cells tests for Python chatbooks is much easier than for Raku chatbooks.
Parsing of Python-Jupyter magic cell arguments is both more restricted and more streamlined than Raku-Jupyter.
In Python it was much easier and more obvious (to me) to figure out how to program creation and usage LLM function objects and make them behave like functions than to implement the Raku LLM-function anonymous (pure, lambda) function solution.
- Actually, it is in my TODO list to have Raku functors; see below.
Copying to clipboard was already implemented in Python (and reliably working) for multiple platforms.
Working Python code is much more often obtained Raku code when using LLMs.
- Hence, Python chatbooks could be seen preferable by some.
My primary use-case was not chatbooks, but finding textual answers in order to re-implement the NLP Template Engine from WL to Raku.
- I have done that to a large degree — see “ML::NLPTemplateEngine”.
- Working on the “workhorse” function llm-find-textual-answer made me look up WRI’s approach to creation of LLM functions and corresponding configurations and evaluators; see [SW1].
Quite a few fragments of this document were created via LLM chats:
- Initial version of the comparison tables from “linear” Markdown lists with links
- The extended primary use formulation
- The narration of the flowchart
I did not just copy-and-pasted the those LLM generated fragments — I read then in full and edited them too!

Future plans

Both

Chatbooks can have magic specs (and corresponding cells) for:
- DeepL
- ProdGDT
A video with comprehensive (long) discussion of multi-cell chats.

Python

Documenting how LLM-generated images can be converted into image objects (and further manipulated image-wise.)

Raku

Make Python chatbooks re-runnable as Raku chatbooks.
- This requires the parsing of Python-style magics.
Implement LLM function objects (functors) in Raku.
- In conjunction of the anonymous functions implementation.
  - Which one is used is specified with an option.
Figure out how to warn users for “almost right, yet wrong” chat cell magic specs.
Implement copy-to-clipboard for Linux and Windows.
- I have put rudimentary code for that, but actual implementation and testing for Linux and Windows are needed.

References

[WRIr1] Wolfram Research Inc., Wolfram Prompt Repository.

[AAp4wl] Anton Antonov, NLPTemplateEngine paclet, (2023) Wolfram Paclet Repository.

Videos

[AAv1] Anton Antonov, “Jupyter Chatbook LLM cells demo (Raku)”, (2023), YouTube/@AAA4Prediction.

[AAv2] Anton Antonov, “Jupyter Chatbook multi-cell LLM chats demo (Raku)”, (2023), YouTube/@AAA4Prediction.

[AAv3] Anton Antonov, “Jupyter Chatbook LLM cells demo (Python)”, (2023), YouTube/@AAA4Prediction.

[AAv4] Anton Antonov, “Jupyter Chatbook multi cell LLM chats teaser (Python)”, (2023), YouTube/@AAA4Prediction.

[AAv5] Anton Antonov, “Simplified Machine Learning Workflows Overview (Raku-centric), (2023), YouTube/@AAA4Prediction.

Jupyter::Chatbook

This blog post proclaims and describes the Raku package “Jupyter::Chatbook” that facilitates the incorporation of Large Language Models (LLMs) into notebooks of Jupyter’s framework.

“Jupyter::Chatbook” is a fork of Brian Duggan’s “Jupyter::Kernel”.

Here are the top opening statements of the README of “Jupyter::Kernel”:

“Jupyter::Kernel” is a pure Raku implementation of a Raku kernel for Jupyter clients¹.

Jupyter notebooks provide a web-based (or console-based) Read Eval Print Loop (REPL) for running code and serializing input and output.

It is desirable to include the interaction with LLMs into the “typical” REPL systems or workflows. Having LLM-aware and LLM-chat-endowed notebooks — chatbooks — can really speed up the:

Writing and preparation of documents on variety of subjects
Derivation of useful programming code
Adoption of programming languages by newcomers

The corresponding repository is mostly for experimental work, but it aims to be always very useful for interacting with LLMs via Raku.

Remark: The reason to have a separate package — a fork of “Jupyter::Kernel” — is because:

I plan to introduce 4-6 new package dependencies
I expect to do a fair amount of UX experimental implementations and refactoring

Installation and setup

From “Zef ecosystem”:

zef install Jupyter::Chatbook

From GitHub:

zef install https://github.com/antononcube/Raku-Jupyter-Chatbook.git

After installing the package “Jupyter::Chatbook” follow the setup instructions of “Jupyter::Kernel”.

Using LLMs in chatbooks

There are four ways to use LLMs in a chatbook:

LLM functions, [AA3, AAp4]
LLM chat objects, [AA4, AAp4]
Code cells with magics accessing LLMs, like, OpenAI’s, [AAp2], or PaLM’s, [AAp3]
Notebook-wide chats that are distributed over multiple code cells with chat-magic specs

The sections below briefly describe each of these ways and have links to notebooks with more detailed examples.

LLM functions and chat objects

LLM functions as described in [AA3] are best utilized via a certain REPL tool or environment. Notebooks are the perfect media for LLM functions workflows. Here is an example of a code cell that defines an LLM function:

use LLM::Functions;

my &fcp = llm-function({"What is the population of the country $_ ?"});

# -> **@args, *%args { #`(Block|5016320795216) ... }

Here is another cell that can be evaluated multiple times using different country names:

<Niger Gabon>.map({ &fcp($_) })

# (
# 
# As of July 2020, the population of Niger is estimated to be 23,843,341. 
# 
# As of July 2019, the population of Gabon is estimated to be 2,210,823 people.)

For more examples of LLM functions and LLM chat objects see the notebook “Chatbook-LLM-functions-and-chat-objects.ipynb”.

LLM cells

The LLMs of OpenAI (ChatGPT, DALL-E) and Google (PaLM) can be interacted with using “dedicated” notebook cells.

Here is an example of a code cell with PaLM magic spec:

%% palm, max-tokens=600
Generate a horror story about a little girl lost in the forest and getting possessed.

For more examples see the notebook “Chatbook-LLM-cells.ipynb”.

Notebook-wide chats

Chatbooks have the ability to maintain LLM conversations over multiple notebook cells. A chatbook can have more than one LLM conversations. “Under the hood” each chatbook maintains a database of chat objects. Chat cells are used to give messages to those chat objects.

For example, here is a chat cell with which a new “Email writer” chat object is made, and that new chat object has the identifier “em12”:

%% chat-em12, prompt = «Given a topic, write emails in a concise, professional manner»
Write a vacation email.

Here is a chat cell in which another message is given to the chat object with identifier “em12”:

%% chat-em12
Rewrite with manager's name being Jane Doe, and start- and end dates being 8/20 and 9/5.

In this chat cell a new chat object is created:

%% chat snowman, prompt = ⎡Pretend you are a friendly snowman. Stay in character for every response you give me. Keep your responses short.⎦
Hi!

And here is a chat cell that sends another message to the “snowman” chat object:

%% chat snowman
Who build you? Where?

Remark: Specifying a chat object identifier is not required. I.e. only the magic spec %% chat can be used. The “default” chat object ID identifier “NONE”.

Remark: The magic keyword “chat” can be separated from the identifier of the chat object with the symbols “-“, “_”, “:”, or with any number of (horizontal) white spaces.

For more examples see the notebook “Chatbook-LLM-chats.ipynb”.

Here is a flowchart that summarizes the way chatbooks create and utilize LLM chat objects:

Chat meta cells

Each chatbook session has a Hash of chat objects. Chatbooks can have chat meta cells that allow the access of the chat object “database” as whole, or its individual objects.

Here is an example of a chat meta cell (that applies the method say to the chat object with ID “snowman”):

%% chat snowman meta
say

Here is an example of chat meta cell that creates a new chat chat object with the LLM prompt specified in the cell (“Guess the word”):

%% chat-WordGuesser prompt
We're playing a game. I'm thinking of a word, and I need to get you to guess that word. 
But I can't say the word itself. 
I'll give you clues, and you'll respond with a guess. 
Your guess should be a single word only.

Here is a table with examples of magic specs for chat meta cells and their interpretation:

cell magic line	cell content	interpretation
chat-ew12 meta	say	Give the “print out” of the chat object with ID “ew12”
chat-ew12 meta	messages	Give the “print out” of the chat object with ID “ew12”
chat sn22 prompt	You pretend to be a melting snowman.	Create a chat object with ID “sn22” with the prompt in the cell
chat meta all	keys	Show the keys of the session chat objects DB
chat all	keys	«same as above»

Here is a flowchart that summarizes the chat meta cell processing:

References

Articles

[AA1] Anton Antonov, “Literate programming via CLI”, (2023), RakuForPrediction at WordPress.

[AA2] Anton Antonov, “Generating documents via templates and LLMs”, (2023), RakuForPrediction at WordPress.

[AA3] Anton Antonov, “Workflows with LLM functions”, (2023), RakuForPrediction at WordPress.

[AA4] Anton Antonov, “Number guessing games: PaLM vs ChatGPT”, (2023), RakuForPrediction at WordPress.

[SW1] Stephen Wolfram, “Introducing Chat Notebooks: Integrating LLMs into the Notebook Paradigm”, (2023), writings.stephenwolfram.com.

Packages

[AAp1] Anton Antonov, Text::CodeProcessing Raku package, (2021), GitHub/antononcube.

[AAp2] Anton Antonov, WWW::OpenAI Raku package, (2023), GitHub/antononcube.

[AAp3] Anton Antonov, WWW::PaLM Raku package, (2023), GitHub/antononcube.

[AAp4] Anton Antonov, LLM::Functions Raku package, (2023), GitHub/antononcube.

[AAp4] Anton Antonov, Text::SubParsers Raku package, (2023), GitHub/antononcube.

[AAp5] Anton Antonov, Data::Translators Raku package, (2023), GitHub/antononcube.

[AAp4] Anton Antonov, Clipboard Raku package, (2023), GitHub/antononcube.

[BDp1] Brian Duggan, Jupyter:Kernel Raku package, (2017-2023), GitHub/bduggan.

Videos

[AAv1] Anton Antonov, “Raku Literate Programming via command line pipelines”, (2023), YouTube/@AAA4Prediction.

[AAv2] Anton Antonov, “Racoons playing with pearls and onions” (2023), YouTube/@AAA4Prediction.

[AAv3] Anton Antonov, “Streamlining ChatGPT code generation and narration workflows (Raku)” (2023), YouTube/@AAA4Prediction.

Footnotes

¹ Jupyter clients are user interfaces to interact with an interpreter kernel like “Jupyter::Kernel”. Jupyter [Lab | Notebook | Console | QtConsole ] are the jupyter maintained clients. More info in the jupyter documentations site.

Text::SubParsers

This blog post proclaims and exemplifies the Raku package “Text::SubParsers” that has functionalities for extracting and processing of interpret-able sub-strings in texts.

The primary motivation for creating this package is the post-processing of the outputs of Large Language Models (LLMs), [AA1, AAp1, AAp2, AAp3].

Installation

From Zef ecosystem:

zef install Text::SubParsers

From GitHub:

zef install https://github.com/antononcube/Raku-Text-SubParsers.git

Usage examples

Date extractions

Here we extract dates from a text:

use Text::SubParsers;
my $res = "Openheimer's birthday is April 22, 1905 or April 2, 1905, as far as I know.";

Text::SubParsers::Core.new('DateTime').subparse($res).raku;

# $["Openheimer's birthday is ", DateTime.new(1905,4,22,0,0,0), " or ", DateTime.new(1905,4,2,0,0,0), ", as far as I know."]

Compare with the result of the parse method over the same text:

Text::SubParsers::Core.new('DateTime').parse($res);

#ERROR: Cannot interpret the given input with the given spec 'DateTime'.
# (Any)

Here are the results of both subparse and parse on string that is a valid date specification:

Text::SubParsers::Core.new('DateTime').subparse('April 22, 1905');

# 1905-04-22T00:00:00Z

Text::SubParsers::Core.new('DateTime').parse('April 22, 1905');

# 1905-04-22T00:00:00Z

Sub-parsing with user supplied subs

Instead of using Text::SubParsers::Core.new the functions get-sub-parser and get-parser can be used.

Here is an example of using:

Invocation of get-sub-parser
(Sub-)parsing with a user supplied function (sub)

sub known-cities(Str $x) { 
    $x ∈ ['Seattle', 'Chicago', 'New York', 'Sao Paulo', 'Miami', 'Los Angeles'] ?? $x.uc !! Nil 
}

get-sub-parser(&known-cities).subparse("
1. New York City, NY - 8,804,190
2. Los Angeles, CA - 3,976,322
3. Chicago, IL - 2,746,388
4. Houston, TX - 2,304,580
5. Philadelphia, PA - 1,608,162
6. San Antonio, TX - 1,5
")

# [
# 1.  NEW YORK  City, NY - 8,804,190
# 2.  LOS ANGELES , CA - 3,976,322
# 3.  CHICAGO , IL - 2,746,388
# 4. Houston, TX - 2,304,580
# 5. Philadelphia, PA - 1,608,162
# 6. San Antonio, TX - 1,5
# ]

Here is the “full form” of the last result

_.raku

# $["\n1. ", "NEW YORK", " City, NY - 8,804,190\n2. ", "LOS ANGELES", ", CA - 3,976,322\n3. ", "CHICAGO", ", IL - 2,746,388\n4. Houston, TX - 2,304,580\n5. Philadelphia, PA - 1,608,162\n6. San Antonio, TX - 1,5\n"]

Sub-parsing with `WhateverCode`

With the parser spec WhateverCode an attempt is made to extract dates, JSON expressions, numbers, and Booleans (in that order). Here is an example:

get-sub-parser(WhateverCode).subparse('
Is it true that the JSON expression {"date": "2023-03-08", "rationalNumber": "11/3"} contains the date 2023-03-08 and the rational number 11/3?
').raku

# $["\nIs it", Bool::True, "that the JSON expression", {:date("2023-03-08"), :rationalNumber("11/3")}, "contains the date", DateTime.new(2023,3,8,0,0,0), "and the rational number", <11/3>, "?\n"]

Processing LLM outputs

As it was mentioned above the primary motivation for creating this package is the post-processing of the outputs of Large Language Models (LLMs), [AA1, AAp1, AAp2, AAp3].

Here is an example of creating a LLM-function and its invocation over a string:

use LLM::Functions;

my &fs = llm-function(
        {"What is the average speed of $_ ?"},
        llm-evaluator => llm-configuration(
                'PaLM',
                prompts => 'You are knowledgeable engineer and you give concise, numeric answers.'));

say &fs('car in USA highway');

# 79.5 mph

Here is the corresponding interpretation using sub-parsers:

get-sub-parser('Numeric').subparse(_.trim).raku;

# $[79.5, "mph"]

Here is a more involved example in which:

A LLM is asked to produce a certain set of events in JSON format
The JSON fragment of the result is parsed
The obtained list of hashes is transformed into Mermaid-JS timeline diagram

my &ft = llm-function(
        {"What are the $^a most significant events of $^b? Give the answer with date-event pairs in JSON format."},
        form => get-sub-parser('JSON'),
        llm-evaluator => llm-configuration('PaLM', max-tokens => 500));

my @ftRes = |&ft(9, 'WWI');
@ftRes = @ftRes.grep({ $_ !~~ Str });

# [{date => 1914-07-28, event => Austria-Hungary declares war on Serbia} {date => 1914-07-29, event => Germany declares war on Russia} {date => 1914-07-30, event => France declares war on Germany} {date => 1914-08-01, event => Great Britain declares war on Germany} {date => 1914-08-04, event => Japan declares war on Germany} {date => 1914-11-09, event => First Battle of Ypres} {date => 1915-05-07, event => Second Battle of Ypres} {date => 1916-07-01, event => Battle of the Somme} {date => 1917-03-08, event => United States declares war on Germany}]

my @timeline = ['timeline', 'title WWI events'];
for @ftRes -> $record {
    @timeline.append( "{$record<date>} : {$record<event>}");
}
@timeline.join("\n\t")

References

Articles

[AA1] Anton Antonov, “LLM::Functions”, (2023), RakuForPrediction at WordPress.

Packages

[AAp1] Anton Antonov, LLM::Functions Raku package, (2023), GitHub/antononcube.

[AAp2] Anton Antonov, WWW::OpenAI Raku package, (2023), GitHub/antononcube.

[AAp3] Anton Antonov, WWW::PaLM Raku package, (2023), GitHub/antononcube.

LLM::Functions

Introduction

The Raku package “LLM::Functions”, [AAp1], provides functions and function objects to access, interact, and utilize Large Language Models (LLMs), like OpenAI, [OAI1], and PaLM, [ZG1].

For more details how the concrete LLMs are accessed see the packages “WWW::OpenAI”, [AAp2], and “WWW::PaLM”, [AAp3].

The primary motivation to have handy, configurable functions for utilizing LLMs came from my work on the packages “ML::FindTextualAnswer”, [AAp5], and “ML::NLPTemplateEngine”, [AAp6].

A very similar system of functionalities is developed by Wolfram Research Inc.; see the paclet “LLMFunctions”, [WRIp1].

For well curated and instructive examples of LLM prompts see the Wolfram Prompt Repository.

The article “Generating documents via templates and LLMs”, [AA1], shows an alternative way of streamlining LLMs usage. (Via Markdown, Org-mode, or Pod6 templates.)

Installation

Package installations from both sources use zef installer (which should be bundled with the “standard” Rakudo installation file.)

To install the package from Zef ecosystem use the shell command:

zef install LLM::Functions

To install the package from the GitHub repository use the shell command:

zef install https://github.com/antononcube/Raku-LLM-Functions.git

Design

“Out of the box” “LLM::Functions” uses “WWW::OpenAI”, [AAp2], and “WWW::PaLM”, [AAp3]. Other LLM access packages can be utilized via appropriate LLM configurations.

Configurations:

Are instances of the class LLM::Functions::Configuration
Are used by instances of the class LLM::Functions::Evaluator
Can be converted to Hash objects (i.e. have a .Hash method)

New LLM functions are constructed with the function llm-function.

The function llm-function:

Has the option “llm-evaluator” that takes evaluators, configurations, or string shorthands as values
Returns anonymous functions (that access LLMs via evaluators/configurations.)
Gives result functions that can be applied to different types of arguments depending on the first argument
Takes as a first argument a prompt that can be a:
- String
- Function with positional arguments
- Function with named arguments

Here is a sequence diagram that follows the steps of a typical creation procedure of LLM configuration- and evaluator objects, and the corresponding LLM-function that utilizes them:

Here is a sequence diagram for making a LLM configuration with a global (engineered) prompt, and using that configuration to generate a chat message response:

Configurations

OpenAI-based

Here is the default, OpenAI-based configuration:

use LLM::Functions;
.raku.say for llm-configuration('OpenAI').Hash;

# :max-tokens(300)
# :tool-response-insertion-function(WhateverCode)
# :prompt-delimiter(" ")
# :evaluator(Whatever)
# :prompts($[])
# :function(proto sub OpenAITextCompletion ($prompt is copy, :$model is copy = Whatever, :$suffix is copy = Whatever, :$max-tokens is copy = Whatever, :$temperature is copy = Whatever, Numeric :$top-p = 1, Int :$n where { ... } = 1, Bool :$stream = Bool::False, Bool :$echo = Bool::False, :$stop = Whatever, Numeric :$presence-penalty = 0, Numeric :$frequency-penalty = 0, :$best-of is copy = Whatever, :$auth-key is copy = Whatever, Int :$timeout where { ... } = 10, :$format is copy = Whatever, Str :$method = "tiny") {*})
# :tool-request-parser(WhateverCode)
# :api-key(Whatever)
# :tools($[])
# :temperature(0.8)
# :module("WWW::OpenAI")
# :total-probability-cutoff(0.03)
# :name("openai")
# :format("values")
# :api-user-id("user:106783131376")
# :tool-prompt("")
# :model("text-davinci-003")
# :stop-tokens($[".", "?", "!"])

Here is the ChatGPT-based configuration:

.say for llm-configuration('ChatGPT').Hash;

# prompts => []
# max-tokens => 300
# total-probability-cutoff => 0.03
# temperature => 0.8
# tool-prompt => 
# api-user-id => user:684744195047
# evaluator => (my \LLM::Functions::ChatEvaluator_5129686912680 = LLM::Functions::ChatEvaluator.new(conf => LLM::Functions::Configuration.new(name => "openai", api-key => Whatever, api-user-id => "user:684744195047", module => "WWW::OpenAI", model => "gpt-3.5-turbo", function => proto sub OpenAIChatCompletion ($prompt is copy, :$type is copy = Whatever, :$role is copy = Whatever, :$model is copy = Whatever, :$temperature is copy = Whatever, :$max-tokens is copy = Whatever, Numeric :$top-p = 1, Int :$n where { ... } = 1, Bool :$stream = Bool::False, :$stop = Whatever, Numeric :$presence-penalty = 0, Numeric :$frequency-penalty = 0, :$auth-key is copy = Whatever, Int :$timeout where { ... } = 10, :$format is copy = Whatever, Str :$method = "tiny") {*}, temperature => 0.8, total-probability-cutoff => 0.03, max-tokens => 300, format => "values", prompts => [], prompt-delimiter => " ", stop-tokens => [".", "?", "!"], tools => [], tool-prompt => "", tool-request-parser => WhateverCode, tool-response-insertion-function => WhateverCode, argument-renames => {:api-key("auth-key")}, evaluator => LLM::Functions::ChatEvaluator_5129686912680)))
# module => WWW::OpenAI
# model => gpt-3.5-turbo
# stop-tokens => [. ? !]
# format => values
# function => &OpenAIChatCompletion
# tool-response-insertion-function => (WhateverCode)
# tool-request-parser => (WhateverCode)
# prompt-delimiter =>  
# api-key => (Whatever)
# tools => []
# name => openai

Remark: llm-configuration(Whatever) is equivalent to llm-configuration('OpenAI').

Remark: Both the “OpenAI” and “ChatGPT” configuration use functions of the package “WWW::OpenAI”, [AAp2]. The “OpenAI” configuration is for text-completions; the “ChatGPT” configuration is for chat-completions.

PaLM-based

Here is the default PaLM configuration:

.say for llm-configuration('PaLM').Hash;

# tools => []
# name => palm
# format => values
# api-user-id => user:157844929178
# prompt-delimiter =>  
# prompts => []
# tool-prompt => 
# total-probability-cutoff => 0
# max-tokens => 300
# temperature => 0.4
# model => text-bison-001
# evaluator => (Whatever)
# api-key => (Whatever)
# stop-tokens => [. ? !]
# function => &PaLMGenerateText
# tool-response-insertion-function => (WhateverCode)
# module => WWW::PaLM
# tool-request-parser => (WhateverCode)

Basic usage of LLM functions

Textual prompts

Here we make a LLM function with a simple (short, textual) prompt:

my &func = llm-function('Show a recipe for:');

# -> $text, *%args { #`(Block|5129723222144) ... }

Here we evaluate over a message:

say &func('greek salad');

# Ingredients:
# 
# -1 head of romaine lettuce, chopped
# -1 large cucumber, diced
# -1 large tomato, diced
# -1/2 red onion, diced
# -1/2 cup kalamata olives, pitted
# -1/2 cup feta cheese, crumbled
# -1/4 cup fresh parsley, chopped
# -2 tablespoons olive oil
# -1 tablespoon red wine vinegar
# -1 teaspoon oregano
# -Salt and pepper, to taste
# 
# Instructions:
# 
# 1. In a large bowl, combine the romaine lettuce, cucumber, tomato, red onion, and kalamata olives.
# 
# 2. Top with feta cheese and parsley.
# 
# 3. In a small bowl, whisk together the olive oil, red wine vinegar, oregano, and salt and pepper.
# 
# 4. Drizzle the dressing over the salad and toss to combine.
# 
# 5. Serve immediately. Enjoy!

Positional arguments

Here we make a LLM function with a function-prompt:

my &func2 = llm-function({"How many $^a can fit inside one $^b?"}, llm-evaluator => 'palm');

# -> **@args, *%args { #`(Block|5129803451072) ... }

Here were we apply the function:

&func2("tenis balls", "toyota corolla 2010");

# (260)

Named arguments

Here the first argument is a template with two named arguments:

my &func3 = llm-function(-> :$dish, :$cuisine {"Give a recipe for $dish in the $cuisine cuisine."}, llm-evaluator => 'palm');

# -> **@args, *%args { #`(Block|5129739321704) ... }

Here is an invocation:

&func3(dish => 'salad', cuisine => 'Russion', max-tokens => 300);

# (**Ingredients**
# 
# * 1 head of cabbage, shredded
# * 1 carrot, shredded
# * 1/2 cup of mayonnaise
# * 1/4 cup of sour cream
# * 1/4 cup of chopped fresh dill
# * 1/4 cup of chopped fresh parsley
# * Salt and pepper to taste
# 
# **Instructions**
# 
# 1. In a large bowl, combine the cabbage, carrots, mayonnaise, sour cream, dill, parsley, salt, and pepper.
# 2. Stir until well combined.
# 3. Serve immediately or chill for later.
# 
# **Tips**
# 
# * To make the salad ahead of time, chill it for at least 30 minutes before serving.
# * For a more flavorful salad, add some chopped red onion or celery.
# * Serve the salad with your favorite bread or crackers.)

Using chat-global prompts

The configuration objects can be given prompts that influence the LLM responses “globally” throughout the whole chat. (See the second sequence diagram above.)

For detailed examples see the documents:

References

Articles

[AA1] Anton Antonov, “Generating documents via templates and LLMs”, (2023), RakuForPrediction at WordPress.

[ZG1] Zoubin Ghahramani, “Introducing PaLM 2”, (2023), Google Official Blog on AI.

Repositories, sites

[OAI1] OpenAI Platform, OpenAI platform.

[WRIr1] Wolfram Research, Inc. Wolfram Prompt Repository.

Packages, paclets

[AAp1] Anton Antonov, LLM::Functions Raku package, (2023), GitHub/antononcube.

[AAp2] Anton Antonov, WWW::OpenAI Raku package, (2023), GitHub/antononcube.

[AAp3] Anton Antonov, WWW::PaLM Raku package, (2023), GitHub/antononcube.

[AAp4] Anton Antonov, Text::CodeProcessing Raku package, (2021), GitHub/antononcube.

[AAp5] Anton Antonov, ML::FindTextualAnswer Raku package, (2023), GitHub/antononcube.

[AAp6] Anton Antonov, ML::NLPTemplateEngine Raku package, (2023), GitHub/antononcube.

[WRIp1] Wolfram Research, Inc. LLMFunctions paclet, (2023), Wolfram Language Paclet Repository.

Menu

Introduction

Themes

Mind-map

Summary, ideas, and recommendations

SUMMARY

IDEAS:

QUOTES:

HABITS:

FACTS:

REFERENCES:

RECOMMENDATIONS:

References

Articles

Packages, repositories

Videos

Introduction

Ways to use with Raku

Packages

Images

CLI

Web API functions

Images

Image narration

Description of a mind-map

LLM Functions

Image ingestion and encoding

Configuration and synthesis

Repeated questioning

Dedicated notebook cells

Combinations (fairytale generation)

References

Articles

Packages

Videos

Introduction

Why doing this?

Article structure

Big picture warm-up

Mind-map

Primary use case

Tabulated comparisons

Primary LLM packages

Secondary LLM packages

Introductory videos

LLM functions

Raku

Python

LLM prompts

Raku

Python

Chatbook cells

Raku

Python

Observations, remarks, and conclusions

Future plans

Both

Python

Raku

References

Articles

Notebooks

Python packages

Raku packages

Wolfram Language paclets

Videos

Installation and setup

Using LLMs in chatbooks

LLM functions and chat objects

LLM cells

Notebook-wide chats

Chat meta cells

References

Articles

Packages

Videos

Footnotes

Installation

Usage examples

Date extractions

Sub-parsing with `WhateverCode`