Create the webapplication logic and visualisations

This module creates the FastHTML webapplication and starts the server.

Stucture of the web-application

The web-application will show on the main page the information flow as created by the create_combined_infoflow_viz from the infoflow.viz module. This function returns a graphviz.graphs.Digraph object. We turn this object into a SVG string which we can then modify to create clickable nodes. To make this possible we need several steps:

  1. Create the infoflow graph, see Create the vizualisation
  2. Convert the graph to an SVG string, see Create the vizualisation
  3. Create a dictionary from all the nodes in the graph
  4. Use that dictionary to alter the SVG string to create clickable nodes
  5. Create the main page from the web-application that shows the information flow as an SVG image.
  6. Create a function to make a webpage for every element in the infoflow.

Ad. 4.

We could also create this dictionary from the instances we made from all the elements in our infoflow. The same instances that are use to create the graph in the first place. But I choose to create the dictionary from the SVG-string from the Digraph object, becaust then I will be able to use that function to create clickable SVG images from other sources as well.

Background information on building the clickable information flow visualisation in a FastHTML web-application

How to visualize a SVG string in a webapp

Use NotStr() to prevent HTML escaping of the SVG string.

svg_sample = """<?xml version="1.0" encoding="UTF-8" standalone="no"?>\n<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"\n "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">\n<!-- Generated by graphviz version 2.43.0 (0)\n -->\n<!-- Title: %3 Pages: 1 -->\n<svg width="226pt" height="534pt"\n viewBox="0.00 0.00 225.61 534.27" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">\n<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 530.27)">\n<title>%3</title>\n<polygon fill="white" stroke="transparent" points="-4,4 -4,-530.27 221.61,-530.27 221.61,4 -4,4"/>\n<!-- neoreader_collect -->\n<g id="node1" class="node">\n<title>neoreader_collect</title>\n<polygon fill="lightblue" stroke="black" points="174.81,-423.24 143.44,-454.3 80.7,-454.3 49.33,-423.24 80.7,-392.19 143.44,-392.19 174.81,-423.24"/>\n<text text-anchor="middle" x="112.07" y="-427.04" font-family="Times,serif" font-size="14.00">NeoReader</text>\n<text text-anchor="middle" x="112.07" y="-412.04" font-family="Times,serif" font-size="14.00">(collect)</text>\n</g>\n<!-- neoreader_retrieve -->\n<g id="node2" class="node">\n<title>neoreader_retrieve</title>\n<polygon fill="orange" stroke="black" points="174.81,-325.19 143.44,-356.24 80.7,-356.24 49.33,-325.19 80.7,-294.13 143.44,-294.13 174.81,-325.19"/>\n<text text-anchor="middle" x="112.07" y="-328.99" font-family="Times,serif" font-size="14.00">NeoReader</text>\n<text text-anchor="middle" x="112.07" y="-313.99" font-family="Times,serif" font-size="14.00">(retrieve)</text>\n</g>\n<!-- neoreader_collect&#45;&gt;neoreader_retrieve -->\n<g id="edge2" class="edge">\n<title>neoreader_collect&#45;&gt;neoreader_retrieve</title>\n<path fill="none" stroke="black" d="M112.07,-392.11C112.07,-384.06 112.07,-375.21 112.07,-366.7"/>\n<polygon fill="black" stroke="black" points="115.57,-366.54 112.07,-356.54 108.57,-366.54 115.57,-366.54"/>\n</g>\n<!-- neoreader_consume -->\n<g id="node3" class="node">\n<title>neoreader_consume</title>\n<polygon fill="lightgreen" stroke="black" points="174.81,-227.13 143.44,-258.19 80.7,-258.19 49.33,-227.13 80.7,-196.08 143.44,-196.08 174.81,-227.13"/>\n<text text-anchor="middle" x="112.07" y="-230.93" font-family="Times,serif" font-size="14.00">NeoReader</text>\n<text text-anchor="middle" x="112.07" y="-215.93" font-family="Times,serif" font-size="14.00">(consume)</text>\n</g>\n<!-- neoreader_retrieve&#45;&gt;neoreader_consume -->\n<g id="edge3" class="edge">\n<title>neoreader_retrieve&#45;&gt;neoreader_consume</title>\n<path fill="none" stroke="black" d="M112.07,-294.06C112.07,-286 112.07,-277.16 112.07,-268.64"/>\n<polygon fill="black" stroke="black" points="115.57,-268.49 112.07,-258.49 108.57,-268.49 115.57,-268.49"/>\n</g>\n<!-- readwise_extract -->\n<g id="node4" class="node">\n<title>readwise_extract</title>\n<polygon fill="lightgreen" stroke="black" points="168.25,-129.08 140.16,-160.13 83.98,-160.13 55.9,-129.08 83.98,-98.03 140.16,-98.03 168.25,-129.08"/>\n<text text-anchor="middle" x="112.07" y="-132.88" font-family="Times,serif" font-size="14.00">Readwise</text>\n<text text-anchor="middle" x="112.07" y="-117.88" font-family="Times,serif" font-size="14.00">(extract)</text>\n</g>\n<!-- neoreader_consume&#45;&gt;readwise_extract -->\n<g id="edge4" class="edge">\n<title>neoreader_consume&#45;&gt;readwise_extract</title>\n<path fill="none" stroke="black" d="M112.07,-196.01C112.07,-187.95 112.07,-179.11 112.07,-170.59"/>\n<polygon fill="black" stroke="black" points="115.57,-170.43 112.07,-160.43 108.57,-170.43 115.57,-170.43"/>\n</g>\n<!-- obsidian_refine -->\n<g id="node5" class="node">\n<title>obsidian_refine</title>\n<polygon fill="lightgreen" stroke="black" points="106.22,-31.03 79.64,-62.08 26.5,-62.08 -0.07,-31.03 26.5,0.03 79.64,0.03 106.22,-31.03"/>\n<text text-anchor="middle" x="53.07" y="-34.83" font-family="Times,serif" font-size="14.00">Obsidian</text>\n<text text-anchor="middle" x="53.07" y="-19.83" font-family="Times,serif" font-size="14.00">(refine)</text>\n</g>\n<!-- readwise_extract&#45;&gt;obsidian_refine -->\n<g id="edge5" class="edge">\n<title>readwise_extract&#45;&gt;obsidian_refine</title>\n<path fill="none" stroke="black" d="M93.57,-97.95C88.34,-89.45 82.59,-80.08 77.09,-71.13"/>\n<polygon fill="black" stroke="black" points="79.93,-69.07 71.72,-62.38 73.97,-72.73 79.93,-69.07"/>\n</g>\n<!-- recall_refine -->\n<g id="node6" class="node">\n<title>recall_refine</title>\n<polygon fill="lightgreen" stroke="black" points="217.65,-31.03 194.36,-62.08 147.78,-62.08 124.49,-31.03 147.78,0.03 194.36,0.03 217.65,-31.03"/>\n<text text-anchor="middle" x="171.07" y="-34.83" font-family="Times,serif" font-size="14.00">Recall</text>\n<text text-anchor="middle" x="171.07" y="-19.83" font-family="Times,serif" font-size="14.00">(refine)</text>\n</g>\n<!-- readwise_extract&#45;&gt;recall_refine -->\n<g id="edge6" class="edge">\n<title>readwise_extract&#45;&gt;recall_refine</title>\n<path fill="none" stroke="black" d="M130.58,-97.95C135.8,-89.45 141.56,-80.08 147.05,-71.13"/>\n<polygon fill="black" stroke="black" points="150.18,-72.73 152.43,-62.38 144.21,-69.07 150.18,-72.73"/>\n</g>\n<!-- source_document -->\n<g id="node7" class="node">\n<title>source_document</title>\n<polygon fill="none" stroke="black" points="149.07,-526.27 75.07,-526.27 75.07,-490.27 149.07,-490.27 149.07,-526.27"/>\n<text text-anchor="middle" x="112.07" y="-504.57" font-family="Times,serif" font-size="14.00">Document</text>\n</g>\n<!-- source_document&#45;&gt;neoreader_collect -->\n<g id="edge1" class="edge">\n<title>source_document&#45;&gt;neoreader_collect</title>\n<path fill="none" stroke="black" d="M112.07,-490.07C112.07,-482.61 112.07,-473.54 112.07,-464.55"/>\n<polygon fill="black" stroke="black" points="115.57,-464.52 112.07,-454.52 108.57,-464.52 115.57,-464.52"/>\n</g>\n</g>\n</svg>\n"""
show(Div(NotStr(svg_sample)))
%3 neoreader_collect NeoReader (collect) neoreader_retrieve NeoReader (retrieve) neoreader_collect->neoreader_retrieve neoreader_consume NeoReader (consume) neoreader_retrieve->neoreader_consume readwise_extract Readwise (extract) neoreader_consume->readwise_extract obsidian_refine Obsidian (refine) readwise_extract->obsidian_refine recall_refine Recall (refine) readwise_extract->recall_refine source_document Document source_document->neoreader_collect

How to make a node from the Digraph SVG-string clickable in FastHTML

  • Add onclick handlers with htmx.ajax calls to the elements you want clickable
  • Include a target div (#content-area) where content will be swapped
  • Add a CSS style attribute so the pointer cursor changes to a hand when hovering over the node

For example you can add this to the element of the node you want clickable:

onclick="htmx.ajax('GET', '/recall-retrieve', {target: '#content-area', swap: 'outerHTML'})"
style="cursor: pointer;"

But a more concise way is to add CSS styling to the FastHTML app that targets all nodes:

.node { cursor: pointer; }

That is what we will be using in this application.

Below is an example of a SVG-string with a clickable node that shows a changing pointer cursor when hovering over it.

svg_sample_click = """<svg width="268pt" height="338pt" viewBox="0.00 0.00 267.84 338.16" xmlns="http://www.w3.org/2000/svg">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 334.16)">
<polygon fill="white" stroke="transparent" points="-4,4 -4,-334.16 263.84,-334.16 263.84,4 -4,4"/>
<g id="node3" class="node" onclick="htmx.ajax('GET', '/recall-retrieve', {target: '#content-area', swap: 'outerHTML'})" style="cursor: pointer;">
<polygon fill="lightgreen" stroke="black" points="198.09,-129.08 163.76,-160.13 95.08,-160.13 60.75,-129.08 95.08,-98.03 163.76,-98.03 198.09,-129.08"/>
<text text-anchor="middle" x="129.42" y="-132.88" font-family="Times,serif" font-size="14.00">Recall</text>
<text text-anchor="middle" x="129.42" y="-117.88" font-family="Times,serif" font-size="14.00">(retrieve)</text>
</g>
</svg>"""

The below example can’t be clicked, because it is not running on a webserver and the GET request will also fail, because the endpoint doesn’t exist yet. But it does show how the pointer changes.

show(Div(NotStr(svg_sample_click), id="content-area"))
Recall (retrieve)

Create a dictionary from all the nodes in the graph

The SVG-string is an XML string. I considered several ways to parse XML string:

  • BeautifulSoup
  • minidom from xml.dom
  • xml.etree.ElementTree

I choose the latest because it is simpler than minidom from xml.dom. The main advantage of xml.dom is that it supports complete DOM operations. But that isn’t needed in this application. BeautifulSoup is propably more forgiving and versatile than xml.etree.ElementTree, but ElementTree is more than enough for this application and does not add another dependency.

We only look for node elements in the root of the XML string. We also skip the edge elements.

source

dict_svgnodes

 dict_svgnodes (svg_str:str)

Example usage of dict_svgnodes()

nds_dict = dict_svgnodes(svg_sample)
pprint(nds_dict)
{'neoreader_collect': {'class': 'node',
                       'fill': 'lightblue',
                       'id': 'node1',
                       'stroke': 'black',
                       'text': ['NeoReader', '(collect)']},
 'neoreader_consume': {'class': 'node',
                       'fill': 'lightgreen',
                       'id': 'node3',
                       'stroke': 'black',
                       'text': ['NeoReader', '(consume)']},
 'neoreader_retrieve': {'class': 'node',
                        'fill': 'orange',
                        'id': 'node2',
                        'stroke': 'black',
                        'text': ['NeoReader', '(retrieve)']},
 'obsidian_refine': {'class': 'node',
                     'fill': 'lightgreen',
                     'id': 'node5',
                     'stroke': 'black',
                     'text': ['Obsidian', '(refine)']},
 'readwise_extract': {'class': 'node',
                      'fill': 'lightgreen',
                      'id': 'node4',
                      'stroke': 'black',
                      'text': ['Readwise', '(extract)']},
 'recall_refine': {'class': 'node',
                   'fill': 'lightgreen',
                   'id': 'node6',
                   'stroke': 'black',
                   'text': ['Recall', '(refine)']},
 'source_document': {'class': 'node',
                     'fill': 'none',
                     'id': 'node7',
                     'stroke': 'black',
                     'text': ['Document']}}

Create clickable nodes in the SVG string

source

add_onclick_to_nodes

 add_onclick_to_nodes (svg_str:str)

Example usage and test

svg_clickable = add_onclick_to_nodes(svg_sample)
svg_clickable
'<?xml version="1.0" encoding="UTF-8" standalone="no"?>\n<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"\n "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">\n<!-- Generated by graphviz version 2.43.0 (0)\n -->\n<!-- Title: %3 Pages: 1 -->\n<svg width="226pt" height="534pt"\n viewBox="0.00 0.00 225.61 534.27" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">\n<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 530.27)">\n<title>%3</title>\n<polygon fill="white" stroke="transparent" points="-4,4 -4,-530.27 221.61,-530.27 221.61,4 -4,4"/>\n<!-- neoreader_collect -->\n<g id="node1" class="node" onclick="htmx.ajax(\'GET\', \'/toolphase?tool=neoreader&phase=collect\', {target: \'#infoflow-graph\', swap: \'outerHTML\'})">\n<title>neoreader_collect</title>\n<polygon fill="lightblue" stroke="black" points="174.81,-423.24 143.44,-454.3 80.7,-454.3 49.33,-423.24 80.7,-392.19 143.44,-392.19 174.81,-423.24"/>\n<text text-anchor="middle" x="112.07" y="-427.04" font-family="Times,serif" font-size="14.00">NeoReader</text>\n<text text-anchor="middle" x="112.07" y="-412.04" font-family="Times,serif" font-size="14.00">(collect)</text>\n</g>\n<!-- neoreader_retrieve -->\n<g id="node2" class="node" onclick="htmx.ajax(\'GET\', \'/toolphase?tool=neoreader&phase=retrieve\', {target: \'#infoflow-graph\', swap: \'outerHTML\'})">\n<title>neoreader_retrieve</title>\n<polygon fill="orange" stroke="black" points="174.81,-325.19 143.44,-356.24 80.7,-356.24 49.33,-325.19 80.7,-294.13 143.44,-294.13 174.81,-325.19"/>\n<text text-anchor="middle" x="112.07" y="-328.99" font-family="Times,serif" font-size="14.00">NeoReader</text>\n<text text-anchor="middle" x="112.07" y="-313.99" font-family="Times,serif" font-size="14.00">(retrieve)</text>\n</g>\n<!-- neoreader_collect&#45;&gt;neoreader_retrieve -->\n<g id="edge2" class="edge">\n<title>neoreader_collect&#45;&gt;neoreader_retrieve</title>\n<path fill="none" stroke="black" d="M112.07,-392.11C112.07,-384.06 112.07,-375.21 112.07,-366.7"/>\n<polygon fill="black" stroke="black" points="115.57,-366.54 112.07,-356.54 108.57,-366.54 115.57,-366.54"/>\n</g>\n<!-- neoreader_consume -->\n<g id="node3" class="node" onclick="htmx.ajax(\'GET\', \'/toolphase?tool=neoreader&phase=consume\', {target: \'#infoflow-graph\', swap: \'outerHTML\'})">\n<title>neoreader_consume</title>\n<polygon fill="lightgreen" stroke="black" points="174.81,-227.13 143.44,-258.19 80.7,-258.19 49.33,-227.13 80.7,-196.08 143.44,-196.08 174.81,-227.13"/>\n<text text-anchor="middle" x="112.07" y="-230.93" font-family="Times,serif" font-size="14.00">NeoReader</text>\n<text text-anchor="middle" x="112.07" y="-215.93" font-family="Times,serif" font-size="14.00">(consume)</text>\n</g>\n<!-- neoreader_retrieve&#45;&gt;neoreader_consume -->\n<g id="edge3" class="edge">\n<title>neoreader_retrieve&#45;&gt;neoreader_consume</title>\n<path fill="none" stroke="black" d="M112.07,-294.06C112.07,-286 112.07,-277.16 112.07,-268.64"/>\n<polygon fill="black" stroke="black" points="115.57,-268.49 112.07,-258.49 108.57,-268.49 115.57,-268.49"/>\n</g>\n<!-- readwise_extract -->\n<g id="node4" class="node" onclick="htmx.ajax(\'GET\', \'/toolphase?tool=readwise&phase=extract\', {target: \'#infoflow-graph\', swap: \'outerHTML\'})">\n<title>readwise_extract</title>\n<polygon fill="lightgreen" stroke="black" points="168.25,-129.08 140.16,-160.13 83.98,-160.13 55.9,-129.08 83.98,-98.03 140.16,-98.03 168.25,-129.08"/>\n<text text-anchor="middle" x="112.07" y="-132.88" font-family="Times,serif" font-size="14.00">Readwise</text>\n<text text-anchor="middle" x="112.07" y="-117.88" font-family="Times,serif" font-size="14.00">(extract)</text>\n</g>\n<!-- neoreader_consume&#45;&gt;readwise_extract -->\n<g id="edge4" class="edge">\n<title>neoreader_consume&#45;&gt;readwise_extract</title>\n<path fill="none" stroke="black" d="M112.07,-196.01C112.07,-187.95 112.07,-179.11 112.07,-170.59"/>\n<polygon fill="black" stroke="black" points="115.57,-170.43 112.07,-160.43 108.57,-170.43 115.57,-170.43"/>\n</g>\n<!-- obsidian_refine -->\n<g id="node5" class="node" onclick="htmx.ajax(\'GET\', \'/toolphase?tool=obsidian&phase=refine\', {target: \'#infoflow-graph\', swap: \'outerHTML\'})">\n<title>obsidian_refine</title>\n<polygon fill="lightgreen" stroke="black" points="106.22,-31.03 79.64,-62.08 26.5,-62.08 -0.07,-31.03 26.5,0.03 79.64,0.03 106.22,-31.03"/>\n<text text-anchor="middle" x="53.07" y="-34.83" font-family="Times,serif" font-size="14.00">Obsidian</text>\n<text text-anchor="middle" x="53.07" y="-19.83" font-family="Times,serif" font-size="14.00">(refine)</text>\n</g>\n<!-- readwise_extract&#45;&gt;obsidian_refine -->\n<g id="edge5" class="edge">\n<title>readwise_extract&#45;&gt;obsidian_refine</title>\n<path fill="none" stroke="black" d="M93.57,-97.95C88.34,-89.45 82.59,-80.08 77.09,-71.13"/>\n<polygon fill="black" stroke="black" points="79.93,-69.07 71.72,-62.38 73.97,-72.73 79.93,-69.07"/>\n</g>\n<!-- recall_refine -->\n<g id="node6" class="node" onclick="htmx.ajax(\'GET\', \'/toolphase?tool=recall&phase=refine\', {target: \'#infoflow-graph\', swap: \'outerHTML\'})">\n<title>recall_refine</title>\n<polygon fill="lightgreen" stroke="black" points="217.65,-31.03 194.36,-62.08 147.78,-62.08 124.49,-31.03 147.78,0.03 194.36,0.03 217.65,-31.03"/>\n<text text-anchor="middle" x="171.07" y="-34.83" font-family="Times,serif" font-size="14.00">Recall</text>\n<text text-anchor="middle" x="171.07" y="-19.83" font-family="Times,serif" font-size="14.00">(refine)</text>\n</g>\n<!-- readwise_extract&#45;&gt;recall_refine -->\n<g id="edge6" class="edge">\n<title>readwise_extract&#45;&gt;recall_refine</title>\n<path fill="none" stroke="black" d="M130.58,-97.95C135.8,-89.45 141.56,-80.08 147.05,-71.13"/>\n<polygon fill="black" stroke="black" points="150.18,-72.73 152.43,-62.38 144.21,-69.07 150.18,-72.73"/>\n</g>\n<!-- source_document -->\n<g id="node7" class="node">\n<title>source_document</title>\n<polygon fill="none" stroke="black" points="149.07,-526.27 75.07,-526.27 75.07,-490.27 149.07,-490.27 149.07,-526.27"/>\n<text text-anchor="middle" x="112.07" y="-504.57" font-family="Times,serif" font-size="14.00">Document</text>\n</g>\n<!-- source_document&#45;&gt;neoreader_collect -->\n<g id="edge1" class="edge">\n<title>source_document&#45;&gt;neoreader_collect</title>\n<path fill="none" stroke="black" d="M112.07,-490.07C112.07,-482.61 112.07,-473.54 112.07,-464.55"/>\n<polygon fill="black" stroke="black" points="115.57,-464.52 112.07,-454.52 108.57,-464.52 115.57,-464.52"/>\n</g>\n</g>\n</svg>\n'
show(Div(NotStr(svg_clickable)))
%3 neoreader_collect NeoReader (collect) neoreader_retrieve NeoReader (retrieve) neoreader_collect->neoreader_retrieve neoreader_consume NeoReader (consume) neoreader_retrieve->neoreader_consume readwise_extract Readwise (extract) neoreader_consume->readwise_extract obsidian_refine Obsidian (refine) readwise_extract->obsidian_refine recall_refine Recall (refine) readwise_extract->recall_refine source_document Document source_document->neoreader_collect 
test(svg_clickable, "Reader", operator.contains) # Check if the node name is in the svg string
test(svg_clickable, "onclick", operator.contains) # Check if the onclick attribute is in the svg string
test(svg_clickable, "#infoflow-graph", operator.contains) # Check if the id is in the svg string
test(svg_clickable, "<svg width=", operator.contains) # Check if the svg tag is in the svg string
test(svg_clickable, "xmlns", operator.contains) # Check if the xmlns attribute is in the svg string