Literate programming via CLI

In this blog post we demonstrate how to do “Literate programming” in Raku via (short) pipelines of Command Line Interface (CLI) scripts.

An alternative of this CLI-based process is to use Mathematica or Jupyter notebooks.

Presentation’s GitHub directory

Video recording


Steps and components

Here is a narration of the diagram above:

  1. Write some text and code in a Markdown file
  2. Weave the Markdown file (i.e. “run it”)
  3. If the woven file the does not have D3.js graphics go to 5
  4. Convert the woven Markdown file into HTML
  5. Examine results
  6. Go to 1
    • Or finish “fiddling with it”

The conversions

The we use the document “Cryptocurrencies-explorations.md” (with Raku code over cryptocurrencies data.)

If no D3.js graphics are specified then we can use the shell command:

file-code-chunks-eval Cryptocurrencies-explorations.md

If D3.js graphics are specified then we can use the shell command:

file-code-chunks-eval Cryptocurrencies-explorations.md && 
  from-markdown Cryptocurrencies-explorations_woven.md -t html -o out.html && 
  open out.html

Remark: It is instructive to examine “Cryptocurrencies-explorations_woven.md” and compare it with “Cryptocurrencies-explorations.md”.

Remark The code chunks with graphics (using “JavaScript::D3”) have to have the chunk option setting results=asis.

Remark The “JavaScript::D3” commands have to have the option settings format => 'html' and div-id => ....


References

Articles

[AA1] Anton Antonov “Raku Text::CodeProcessing”, (2021), RakuForPrediction at WordPress.

[AA2] Anton Antonov “JavaScript::D3”, (2022), RakuForPrediction at WordPress.

[AA3] Anton Antonov “Further work on the Raku-D3.js translation”, (2022), RakuForPrediction at WordPress.

Packages

[AAp1] Anton Antonov, Data::Cryptocurrencies Raku package, (2023). GitHub/antononcube.

[AAp2] Anton Antonov, JavaScript::D3 Raku package, (2022). GitHub/antononcube.

[AAp3] Anton Antonov, Text::CodeProcessing Raku package, (2021). GitHub/antononcube.

[AAp4] Anton Antonov, Markdown::Grammar, (2022). GitHub/antononcube.

Data::Cryptocurrencies

The Raku package “Data::Cryptocurrencies” has functions for cryptocurrency data retrieval. (At this point, only Yahoo Finance is used as a data source.)

The implementation follows the Mathematica implementation in [AAf1] described in [AA1]. (Further explorations are discussed in [AA2].)


Installation

From Zef ecosystem:

zef install Data::Cryptocurrencies

From GitHub:

zef install https://github.com/antononcube/Raku-Data-Cryptocurrencies.git


Usage examples

Here we get Bitcoin (BTC) data from 1/1/2020 until now:

use Data::Cryptocurrencies;
use Data::Summarizers;
use Text::Plot;

my @ts = cryptocurrency-data('BTC', dates => (DateTime.new(2020, 1, 1, 0, 0, 0), now), props => <DateTime Close>,
        format => 'dataset'):!cache-all;

say @ts.elems;

# 1137

When we request the data to be returned as “dataset” then the result is an array of hashes. When we request the data to be returned as “timeseries” the result is an array of pairs (sorted by date.) 

Here are BTC values for the last week (at the point of retrieval):

.say for @ts.tail(7)

# {Close => 23331.847656, DateTime => 2023-02-04T00:00:00Z}
# {Close => 22955.666016, DateTime => 2023-02-05T00:00:00Z}
# {Close => 22760.109375, DateTime => 2023-02-06T00:00:00Z}
# {Close => 23264.291016, DateTime => 2023-02-07T00:00:00Z}
# {Close => 22939.398438, DateTime => 2023-02-08T00:00:00Z}
# {Close => null, DateTime => 2023-02-09T00:00:00Z}
# {Close => 21863.925781, DateTime => 2023-02-10T00:00:00Z}

Here is a summary:

records-summary(@ts, field-names => <DateTime Close>);

# +--------------------------------+----------------------+
# | DateTime                       | Close                |
# +--------------------------------+----------------------+
# | Min    => 2020-01-01T00:00:37Z | 8755.246094  => 1    |
# | 1st-Qu => 2020-10-10T12:00:37Z | 45585.03125  => 1    |
# | Mean   => 2021-07-22T00:00:37Z | 55888.132813 => 1    |
# | Median => 2021-07-22T00:00:37Z | 29098.910156 => 1    |
# | 3rd-Qu => 2022-05-02T12:00:37Z | 20471.482422 => 1    |
# | Max    => 2023-02-10T00:00:37Z | 21395.019531 => 1    |
# |                                | 53555.109375 => 1    |
# |                                | (Other)      => 1130 |
# +--------------------------------+----------------------+

Clean data:

@ts = @ts.grep({ $_<Close> ~~ Numeric }).Array;
say @ts.elems;

# 1136

Here is a text-based plot of the corresponding time series:

say text-list-plot(@ts.map(*<DateTime>.Instant.Int).List, @ts.map(*<Close>).List, width => 100, height => 20);

# +------+-----------------+-----------------+-----------------+-----------------+-----------------+-+          
# +                                                                                                  +  70000.00
# |                                                       * *                                        |          
# |                                      *  *             ***                                        |          
# +                                     *******           *****                                      +  60000.00
# |                                     *******        * **  **                                      |          
# +                                    **** ***       ** *    ***                                    +  50000.00
# |                                    **     *      *****    ***  *   *                             |          
# |                                 *         **    **  **      ** *******                           |          
# +                                 ****      ****  *             ***** ***                          +  40000.00
# |                                 ***        ******             *       *                          |          
# +                                 ***          * *                      ***                        +  30000.00
# |                                **                                        *   *                   |          
# |                                *                                         ********* **    ***     |          
# +                             ***                                          ***  ************       +  20000.00
# |                      *    ***                                                       **           |          
# +     ******   **************                                                                      +  10000.00
# |    **    *****                                                                                   |          
# +                                                                                                  +      0.00
# +------+-----------------+-----------------+-----------------+-----------------+-----------------+-+          
#        1580000000.00     1600000000.00     1620000000.00     1640000000.00     1660000000.00     1680000000.00

Instead of the text plot above we can use “JavaScript::D3”:


Data caching

Since the downloading of the cryptocurrencies data can be time consuming ( close to 1 minute) it is a good idea to do data caching.

Data caching is “triggered” with the adverb :cache-all. Here is an example:

cryptocurrency-data('BTC', dates => 'today'):cache-all;

By default no data caching is done (i.e. :!cache-all.) The data is stored in the $XDG_DATA_HOME directory. See [JS1]. Every day new data is obtained.

Remark: The use of cached data greatly speeds up the cryptocurrencies explorations with this package.


CLI

The package provides a Command Line Interface (CLI) script. Here is its usage message:

cryptocurrency-data --help

# Usage:
#   cryptocurrency-data [<symbol>] [-p|--properties=<Str>] [--start-date=<Str>] [--end-date=<Str>] [-c|--currency=<Str>] [--format=<Str>] -- Retrieves cryptocurrency data.
#   
#     [<symbol>]               Cryptocurrency symbol. [default: 'BTC']
#     -p|--properties=<Str>    Properties to retrieve. [default: 'all']
#     --start-date=<Str>       Start date. [default: 'auto']
#     --end-date=<Str>         End date. [default: 'now']
#     -c|--currency=<Str>      Currency. [default: 'USD']
#     --format=<Str>           Format of the result [default: 'json']


Additional usage examples

The notebook “Cryptocurrency-explorations.ipynb” provides additional usage examples using D3.js plots. (It loosely follows [AA2].)


References

Articles

[AA1] Anton Antonov “Crypto-currencies data acquisition with visualization” , (2021), MathematicaForPrediction at WordPress.

[AA2] Anton Antonov “Cryptocurrencies data explorations” , (2021), MathematicaForPrediction at WordPress.

Functions, packages

[AAf1] Anton Antonov, CryptocurrencyData Mathematica resource function, (2021), WolframCloud/antononcube.

[AAp1] Anton Antonov, Data::Summarizers Raku package, (2021-2023), GitHub/antononcube.

[AAp2] Anton Antonov, Text::Plot Raku package, (2021), GitHub/antononcube.

[JS1] Jonathan Stowe, XDG::BaseDirectory Raku package, (2016-2023), Zef-ecosystem/jonathanstowe.

Further work on the Raku-D3.js translation

Introduction

This document gives motivation and design details for the Raku package “JavaScript::D3” and discusses the usefulness of implementing:

  • The package itself
  • The corresponding Python package “JavaScript::D3”, [AAp3]
  • Functions for drawing random mandalas and scribbles

Some familiarity with “JavaScript::D3” is assumed. See the article “JavaScript::D3”, or the video “The Raku-ju hijack hack of D3.js”, [AAv1].


Why I implement this package?

Using D3.js via Raku is a good answer of the question “How to do data-driven visualizations with Raku?”

I used to (or tried to use) the alternatives outlined in the following sub-sections.

SVG::Plot

The package “SVG::Plot” is a good illustration that a Raku visualization system can be build “from scratch”, but it is way too much work to develop “SVG::Plot” to have all features needed for informative plots. (Various plot labels, grid lines, etc.)

One of the best features of “SVG::Plot” is that it is supported “out of the box” by the Jupyter Raku kernel [BD1].

Chart::Gnuplot

I have attempted to install the package “Chart::Gnuplot” a few times on my computer without success. Some people from the Raku community claim it is useful.

Text::Plot

I wrote earlier this year the package “Text::Plot”, because I needed some plotting for my presentation at TRC-2022.

Its main advantage is that “it can be used anywhere.” Occasionally, I find it very useful, but it is just a crutch, not a real plotting solution.

For example, it is hard to have informative time series plotted with “Text::Plot”. In support of that statement see this time series plot done with Mathematica for the number of IRC Raku channel posts per week in the last three years (by certain known posters):

That is why!

When I do data analysis I want to be able to:

  1. Make nice and informative plots
  2. Use concise plot specifications
  3. Quickly do the required setup
  4. Claim easy reproducibility and portability of the related documents

Using D3.js definitely provides 1. The package “JavaScript::D3” aims to provide 2. Setting up Jupyter (notebooks) with the Raku kernel implemented by Brian Duggan, [BD1], is not that hard. (But be warned that YMMV.) The Jupyter notebooks are widely shareable at this point, so, 4 is also satisfied.

Here is an example bubble plot made in Jupyter:


Why I did not implement this a year ago?

I do not like Jupyter much, and I mostly stay away from it. That is why, 1.5 years ago when I read the code of “Jupyter::Kernel” I did not try to comprehend the magics. (I was interested in sand-boxing Raku.)

Brian Duggan has put examples how the magics can be used to embed HTML and SVG code in Jupyter, [BD1]. Those prompted me to utilize that framework to use JavaScript.


Refactoring via re-implementation in Python

Re-implementing the package (from Raku) to Python was a great way to brainstorm the necessary refactoring of the JavaScript code snippets and related package architecture. See the examples in [AAp3].


Multi-dataset visualizations

The initial version of “JavaScript::D3” did not have multi-dataset support for bar charts and histograms. I figured out how to do that for bar charts; still working on histograms.

Here is a bar chart example:



Random mandalas and scribbles

Another way to “challenge” the package design and implementation is to (try to) implement functions that draw random mandalas, [AAv2], and random scribbles.

Implementing the corresponding functions required to:

  1. Have optional axes specification
  2. Be able to produce D3.js without placement wrappers (like HTML declaration, of JavaScript brackets for Jupyter.)

The refactoring for 2. was required in order to have multiple plots in one Jupyter cell. Also, it was fairly involved, although, it is “just” for gluing the main, core plotting parts.

I wanted to do as much as possible on the Raku side:

  • The random points generation and “arrangement” is done in Raku
  • The plotting is done with D3.js

Remark: The algorithms for drawing, say, Bezier curves through the random points are far from trivial and D3.js does this very nicely.

Here is an example of random mandala generation:

Here is an example of random scribbles generation:

For interactive examples, see the video “Random mandalas generation (with D3.js via Raku)”.


Some “leftover” remarks

  • My Jupyter setup is “local” on my laptop. I have tried to use Raku in cloud setups a few times without much success.
  • I think there are some issues with the ZMQ connections between Raku’s “Jupyter::Kernel” and “the big” Jupyter framework. Observing those difficulties and making reproducible bug reports is not easy.
  • If more people use Raku with Jupyter that would help or induce reliability improvements.
  • (I hope my demos with “JavaScript::D3” would make some Rakunistas interrupt their VI addiction and try the Raku-Jupyter synergy.)

References

Articles

[AA1] Anton Antonov, “JavaScript::D3”, (2022), RakuForPrediction at WordPress.

[OV1] Olivia Vane, “D3 JavaScript visualisation in a Python Jupyter notebook”, (2020), livingwithmachines.ac.uk.

[SF1] Stefaan Lippens, Custom D3.js Visualization in a Jupyter Notebook, (2018), stefaanlippens.net.

Packages

[AAp1] Anton Antonov, Data::Reshapers Raku package, (2021-2022), GitHub/antononcube.

[AAp2] Anton Antonov, Text::Plot Raku package, (2022), GitHub/antononcube.

[AAp3] Anton Antonov, JavaScriptD3 Python package, (2022), Python-packages at GitHub/antononcube.

[BD1] Brian Duggan, Jupyter::Kernel Raku package, (2017-2022), GitHub/bduggan.

[MLp1] Moritz Lenz, SVG::Plot Raku package (2009-2018), GitHub/moritz.

Videos

[AAv1] Anton Antonov, “The Raku-ju hijack hack of D3.js”, (2022), Anton Antonov’s channel at YouTube.

[AAv2] Anton Antonov, “Random mandalas generation (with D3.js via Raku)”, (2022), Anton Antonov’s channel at YouTube.