The Joy of Tagging: Manipulating and Mining Notebooks in Mathematica

David Reiss

Scientific ArtsLLC

This conference notebook available from

http://scientificarts.com/worklife/conference2008

Basics

Mathematica's Notebook interface is one of the least utilized aspects of Mathematica

A Key tool in building document-oriented application: Tagging

General definition: Tags are any Cell or Notebook contents—generally non-visible—used to navigate, manipulate, mine, or embed material in a Notebook

Tools and tricks that I have learned and created to build these applications

Tagging allows you to do... A Huge Number of Things

Touch on a few of the learned lessons in this talk: but it will get you started

Quick History: The Evolution of Notebooks

Everything is an expression...  in Mathematica

For some tricks and ideas take a look at "Every Document is an Expression: The Reach of Mathematica 6" at:

http://www.scientificarts.com/worklife/notebooks/

It hasn't always been that way....

TextCell[
Button["A Version 2.1 Notebook Snippit",
NotebookPut[Notebook[{
Cell[TextData[{"Part of the internal contents of a ",StyleBox["Mathematica",FontSlant->"Italic"]," version2.1 Notebook"}],"Subsection",Editable->True,Deletable->True,CellDingbat->None,ShowCellBracket->False,CellChangeTimes->{{3.400621354593155*^9,3.400621380018284*^9}},TextAlignment->Center],

Cell[BoxData[" "],"Input",ShowCellBracket->False,CellMargins->{{0,0},{0,0}},CellChangeTimes->{3.400621397349637*^9},FontSize->2,Background->GrayLevel[0.66],CellID->1899068632],

Cell["\<(*^
  
  ::[\tfrontEndVersion = \"Macintosh Mathematica Notebook Front End Version 2.1\";
    \tmacintoshStandardFontEncoding; 
    \tpaletteColors = 128; automaticGrouping; currentKernel; 
    \tfontset = title, inactive, noPageBreakBelow, nohscroll, preserveAspect, groupLikeTitle, center, M7, bold, e8,  24, \"Times\"; ;
    \tfontset = subtitle, inactive, noPageBreakBelow, nohscroll, preserveAspect, groupLikeTitle, center, M7, bold, e6,  18, \"Times\"; ;
    \tfontset = subsubtitle, inactive, noPageBreakBelow, nohscroll, preserveAspect, groupLikeTitle, center, M7, italic, e6,  14, \"Times\"; ;
    \tfontset = section, inactive, noPageBreakBelow, nohscroll, preserveAspect, groupLikeSection, grayBox, M22, bold, a20,  18, \"Times\"; ;
    \tfontset = subsection, inactive, noPageBreakBelow, nohscroll, preserveAspect, groupLikeSection, blackBox, M19, bold, a15,  14, \"Times\"; ;
    \tfontset = subsubsection, inactive, noPageBreakBelow, nohscroll, preserveAspect, groupLikeSection, whiteBox, M18, bold, a12,  12, \"Times\"; ;
    \tfontset = text, inactive, nohscroll, noKeepOnOnePage, preserveAspect, M7,  12, \"Times\"; ;
    \tfontset = smalltext, inactive, nohscroll, noKeepOnOnePage, preserveAspect, M7,  10, \"Times\"; ;
    \tfontset = input, noPageBreakInGroup, nowordwrap, preserveAspect, groupLikeInput, M42, N23, bold, L - 5,  12, \"Courier\"; ;
    \tfontset = output, output, inactive, noPageBreakInGroup, nowordwrap, preserveAspect, groupLikeOutput, M42, N23, L - 5,  12, \"Courier\"; ;
    \tfontset = message, inactive, noPageBreakInGroup, nowordwrap, preserveAspect, groupLikeOutput, M42, N23, R65535, L - 5,  12, \"Courier\"; ;
    \tfontset = print, inactive, noPageBreakInGroup, nowordwrap, preserveAspect, groupLikeOutput, M42, N23, L - 5,  12, \"Courier\"; ;
    \tfontset = info, inactive, noPageBreakInGroup, nowordwrap, preserveAspect, groupLikeOutput, M42, N23, B65535, L - 5,  12, \"Courier\"; ;
    \tfontset = postscript, PostScript, formatAsPostScript, output, inactive, noPageBreakInGroup, nowordwrap, preserveAspect, groupLikeGraphics, M7, l34, w282, h287,  12, \"Courier\"; ;
    \tfontset = name, inactive, nohscroll, noKeepOnOnePage, preserveAspect, M7, italic,  10, \"Geneva\"; ;
    \tfontset = header, inactive, noKeepOnOnePage, preserveAspect, M7,  12, \"Times\"; ;
    \tfontset = leftheader, inactive, L2,  12, \"Times\"; ;
    \tfontset = footer, inactive, noKeepOnOnePage, preserveAspect, center, M7,  12, \"Times\"; ;
    \tfontset = leftfooter, inactive, L2,  12, \"Times\"; ;
    \tfontset = help, inactive, nohscroll, noKeepOnOnePage, preserveAspect, M7,  10, \"Times\"; ;
    \tfontset = clipboard, inactive, nohscroll, noKeepOnOnePage, preserveAspect, M7,  12, \"Times\"; ;
    \tfontset = completions, inactive, nohscroll, noKeepOnOnePage, preserveAspect, M7,  12, \"Times\"; ;
    \tfontset = special1, inactive, nohscroll, noKeepOnOnePage, preserveAspect, M7,  12, \"Times\"; ;
    \tfontset = special2, inactive, nohscroll, noKeepOnOnePage, preserveAspect, M7,  12, \"Times\"; ;
    \tfontset = special3, inactive, nohscroll, noKeepOnOnePage, preserveAspect, M7,  12, \"Times\"; ;
    \tfontset = special4, inactive, nohscroll, noKeepOnOnePage, preserveAspect, M7,  12, \"Times\"; ;
    \tfontset = special5, inactive, nohscroll, noKeepOnOnePage, preserveAspect, M7,  12, \"Times\"; ;
    ]
 : [font = input; preserveAspect; ]
  << vertflux14.m
    
    : [font = input; preserveAspect; startGroup; ]
     << KE_Final8.m
       : [font = message; inactive; preserveAspect; ]
        earthRadiusScale::shdw : 
             Warning : Symbol earthRadiusScale appears in multiple contexts {knife`, Vertflux`}
    ; definitions in context knife` may shadow or be shadowed by other definitions.
 : [font = message; inactive; preserveAspect; endGroup; ]
  dB::shdw : Warning : Symbol dB appears in multiple contexts {knife`, Vertflux`}
    ; definitions in context knife` may shadow or be shadowed by other definitions.
  : [font = input; preserveAspect; ]
   << KE_Final8.m
     : [font = input; preserveAspect; ]
      field2V::usage = \"field2V[transmitterHeight, targetHeight, distance, phi] is the 
 vertical electromagnetic field relative to free space at the target. 
 The transmitterHeight and targetHeight are heights above the earths surface in meters. 
 distance is the distance along the earths surface from the transmitter to the target also 
 measured in meters. phi is the frequency measured in mhz. field2V[transmitterHeight, 
 targetHeight, distance, phi, {sigma, epsilon}] uses sigma and epsilon as the earths 
 ground parameters (the defaults are 0.005 and 15 respectively). field2V[transmitterHeight, targetHeight, distance, phi, nterms] uses nterms in the sum (the default is 5).\"\>","SR",Editable->True,Magnification->0.75,CellID->856781515]
}]],Appearance->"Palette",BaseStyle->{FontSize->12,FontFamily->"Helvetica"}],
"Text",TextAlignment->Center,ShowStringCharacters->False]//CellPrint

"A Version 2.1 Notebook Snippet"

Then it got better (Versions 3⟶5.x):

NotebookPut[
Notebook[{
Cell["An integral to do","Subsection"],
Cell[BoxData[
ToBoxes[Unevaluated[Integrate[(x/(1+ Sin[x])),x]]]],"Text"],
Cell["And its result","Subsection"],
Cell[BoxData[
ToBoxes[Integrate[(x/(1+ Sin[x])),x]]],"Output",CellTags->{"tag1"}]
}]];

And better still  (Versions 6.x+)

nbTemp=CreateDocument[
{
TextCell["An integral to do","Subsection"],
ExpressionCell[Defer@Integrate[(x/(1+ Sin[x])),x],"Text"],
TextCell["And its result","Subsection"],
ExpressionCell[Integrate[(x/(1+ Sin[x])),x],"Output",CellTags->{"tag1"}]
}];

Simple example of the use of CellTags to extract information

Module[{nbr,theNotebook},
(* find the cells with the tag *)
NotebookFind[nbTemp,"tag1",All,CellTags]; 
(* read those cells *)
nbr= NotebookRead[nbTemp]; 
(* prepare the notebook expression *)
theNotebook= Notebook[Flatten[{nbr}]]; 
(* and open the notebook *)
NotebookPut[theNotebook] 
(* create a new notebook with those cells *)
];

Mathematica's Tagging Interface:

Mathematica has wonderful set of tagging functions, but a feature-poor interface for tagging

And some functions are not well documented or documented at all

Cell>CellTags>Add/Remove CellTags  menu item for adding and removing tags

TextCell[Button["Mathematica Cell Tags Interface",FrontEndTokenExecute["CellTagsEditDialog"], Appearance->"Palette",BaseStyle->{FontSize->12,FontFamily->"Helvetica"}],
"Text",TextAlignment->Center,ShowStringCharacters->False]//CellPrint

"Mathematica Cell Tags Interface"

Cell>CellTags>Find Cell Tag  menu item for finding all cells with a given tag

But,... with the right tricks one can make a more useful set of tools:

Tagging Palette Image

Tagging in a Nutshell: Types, Tools, and Tips

A Compressed List: We'll Cover Examples of These in this Talk

Conventional

CellTags

CellID [New in 6]

Requires that the notebook have System`CreateCellID->True: Undocumented but here to stay (according to John Fultz).  The Documentation Center uses CellIDs extensively.

Generalized

TaggingRules

CellChangeTimes

CellTags with embedded information

CellEventActions and NotebookEventActions

Tools

NotebookFind, NotebookLocate, NotebookRead, NotebookWrite, NotebookGet, SelectionMove

NotebookObject, NotebookSelection

RandomCellTag, NotebookFindGeneralized (will define these later), NotebookFindCellTagComplement (from A WorkLife FrameWork: will explain how to build it yourself)

Scratch Notebooks (generally with Visible->False)

When you want to work on material from a notebook do it here

Necessary for using FrontEndTokens on material

FrontEndTokens with FrontEndExecute when convenient

Compressed Data (for storing things in tags)

AuthorTools (for useful access to low level information)

Tips

Use quick tagging for CellTags via MathLink`CallFrontEnd for the functions

FrontEnd`GetCellTagsPacket

FrontEnd`SelectionAddCellTags

FrontEnd`SelectionRemoveCellTags

CreateDocument@(
MathLink`CallFrontEnd[FrontEnd`GetCellTagsPacket[EvaluationNotebook[]]]
);

Gathering CellTags Cell-by-Cell is very slow compared to this. (Example to follow later on...)

Things You Can Do With Tags: From A WorkLife FrameWork

Just Two Examples Out of Many Possible

ToDos

NotebookOpen[ToFileName[DirectoryName@CurrentValue["NotebookDirectory"],"ToDosExample.nb"]];
ToDosPalette[];

Cell Backups

CreateDocument[TextCell["This is a cell that has not yet been backed up.","Text"]];
FormattingPalette[];

Example of an Enhanced Tagging Interface: Tagging Palette from A WorkLife FrameWork

These are tools from A WorkLife FrameWork,...  but you can make your own.  Tricks follow...

Function[x,
TextCell[Button["Typical Notebook",
(nbTemp=NotebookPut[x];
SetOptions[nbTemp,DockedCells->{}];
TaggingPalette[]),
Appearance->"Palette",BaseStyle->{FontSize->12,FontFamily->"Helvetica"}],
"Text",TextAlignment->Center,ShowStringCharacters->False,GeneratedCell->False,CellAutoOverwrite->False]
][Import[ToFileName[CurrentValue["NotebookDirectory"],"ConferenceTalkExample.nb"],"NB"]]//CellPrint

"Typical Notebook"

The parameter nbTemp is assigned to the NotebookObject created from the button above

TaggingPalette[];

A example of non-linear extraction: an antidote to sloppy (i.e., Creative) work habits

This example uses FrontEndTokens

NotebookFind[nbTemp,"Final",All,CellTags];

FrontEndExecute[FrontEndToken[nbTemp,"Copy"]];

FrontEndExecute[FrontEndToken[CreateWindow[Sequence@@Options[nbTemp]],"Paste"]]

Some Tools I

RandomTag

RandomTag[]:=ToString[RandomInteger[{(10^9),(10^10)-1}]];

AddTags

AddTags[nb_NotebookObject,{tags__}]:=MathLink`CallFrontEnd[FrontEnd`SelectionAddCellTags[nb,{tags}]];

RemoveTags

RemoveTags[nb_NotebookObject,{tags__}]:=MathLink`CallFrontEnd[FrontEnd`SelectionRemoveCellTags[nb,{tags}]];

CellTagsInNotebook

CellTagsInNotebook[nb_NotebookObject]:=MathLink`CallFrontEnd[FrontEnd`GetCellTagsPacket[nb]];

Some Tools I... Continued: Speed Comparison

Fast Vs Slow

For this notebook:

Fast: use the FrontEnd` function

time1=First[Timing[CellTagsInNotebook[EvaluationNotebook[]];//Timing]]

Slow: grab the whole notebook and get the tags using pattern matching

time2=
First[
Timing[
Module[{theNotebook,cells},
theNotebook=NotebookGet[EvaluationNotebook[]];
cells=First[theNotebook];

Cases[cells,Cell[_,___,CellTags->z_,___]:>z,∞]
]
]
]

time2/time1

For another, large, notebook (the development notebook for A WorkLife FrameWork)

Fast

time1=First[Timing[CellTagsInNotebook[$CurrentPackageNotebook];//Timing]]

Slow

time2=
First[
Timing[
Module[{theNotebook,cells},
theNotebook=NotebookGet[$CurrentPackageNotebook];
cells=First[theNotebook];

Cases[cells,Cell[_,___,CellTags->z_,___]:>z,∞]
]
]
]

time2/time1

Some Tools II

Generalization of NotebookFind to Multiple Tags

Find the cells with the tags/data and apply a unique tag to them

Find all cells with that unique tag

[Housekeeping] Remove the unique tag from those cells

NotebookFindGeneralized[nb_NotebookObject,data:{__},z___,opts:OptionsPattern[]]:=
Module[{guid,nbf},

guid=RandomTag[];

(
NotebookFind[nb,#,z,AutoScroll->False,opts];
SelectionMove[nb,All,Cell,AutoScroll->False];
AddTags[nb,{guid}];
)&/@data;

nbf=NotebookFind[nb,guid,All,CellTags,opts];
SelectionMove[nb,All,Cell,AutoScroll->False];
RemoveTags[nb,{guid}];

nbf
];

Example of its use

Create an example notebook

nbTemp=
CreateDocument[
{
TextCell["Simple integration example","Section"],

TextCell["Here is an integral","Subsection"],

ExpressionCell[Defer@Integrate[(x/(1+ Sin[x])),x],"Input",CellTags->{"expression"}],

TextCell["This is the result","Subsection"],

TextCell[Integrate[(x/(1+ Sin[x])),x],"Output",CellTags->{"result"}]
}];

Extract content with multiple CellTags as well as CellContents

Module[{nbr},

NotebookFindGeneralized[nbTemp,
{"expression","result"},All,{CellTags,CellContents}];

nbr=NotebookRead[nbTemp];

NotebookPut[Notebook[Flatten[{nbr}]]]
];

NotebookFind Using Regular Expressions for CellTags

Recipe

With CellTagsInNotebook and NotebookFindGeneralized  you can find cells based on RegularExpressions

Find all of the CellTags in the notebook using CellTagsInNotebook

Select the tags you want from the list using StringMatchQ and RegularExpression

Then use NotebookFindGeneralized to select the cells with the list of tags that StringMatchQ returns

Finding the Complement of a tag

Steps

Select all cells in the notebook

Apply a unique tag to them

Find all cells with the tag to complement

Remove the unique tag from those cells

Find all cells with the unique tag: these are the Complement

[Housekeeping] Remove the unique tag from those cells

From A WorkLife FrameWork:

?NotebookFindCellTagComplement

This is the function behind the buttons that are preceded by a "!" in A WorkLife FrameWork's TaggingPalette

Tagging Palette Image

CellID

Note that CreateCellID is not properly declared in the System` Context in either Version 6 or 7.  Always specify it as System`CreateCellID

CellID:

Automatically created if the notebook has the option System`CreateCellID->True

Automatically assigning CellIDs is not the default behavior of a notebook in version 6

Each Cell in a notebook is assigned a unique CellID

Pasting the same Cell in a notebook several times creates copies of the cell with distinct CellIDs

If a notebook's cells do not have CellIDs then you can update that notebook by:

Copying the whole notebook

Set the notebook's options so that System`CreateCellID -> True

Paste the notebook back into itself

Here is a function that does this

NotebookCellIDsUpdate[nb_NotebookObject]:=
Module[{nbg,editable},

editable=Editable/.Options[nb,Editable];

SetOptions[nb,Editable->True];

SetOptions[nb,System`CreateCellID->True];

nbg=NotebookGet[nb];

SelectionMove[nb,All,Notebook];
SetOptions[NotebookSelection[nb],Editable->True,Deletable->True,Copyable->True];

NotebookPut[nbg,nb];

SetOptions[nb,Editable->editable]

];

Example

nbTemp=CreateDocument[
TextCell["A cell with a CellID","Text",ShowCellBracket->True]
];

NotebookCellIDsUpdate[nbTemp]

Finding Cells by CellID

NotebookFind can use CellIDs

NotebookFind[EvaluationNotebook[],PutCellIDHere,All,CellID]

CellID Acts Like an HTML Anchor Tag in a Paclet URI: Documentation Center

Two examples with the Documentation Center

Tooltip[
Button["Open Link",FrontEndExecute[
   FrontEndToken["OpenHelpLink", {"paclet:tutorial/IntroductionToDynamic#962859965"}]]],
"Slow Evaluations inside Dynamic"]

TextCell[Text@Hyperlink["Slow Evaluations inside Dynamic", "paclet:tutorial/IntroductionToDynamic#962859965"],"Text"]//CellPrint

MouseOver

Relating CellID and CellTags

Function for retrieving the Notebook Outline

Needs["AuthorTools`Common`"]

AuthorTools still has a trove of things to make use of or to crib for your own applications!

?NotebookFileOutline

NotebookFileOutline requires that the notebook has been saved to disk. Also it only returns that information that was in the notebook at the time of last saving.

"Typical Notebook"

exampleFileOutline=NotebookFileOutline[nbTemp];

cellIDCellTagsInfo=Cases[exampleFileOutline,Cell[___,(CellID->x_),___,(CellTags->y_)]|Cell[___,(CellTags->y_),___,(CellID->x_)]:>(x->y),∞];

CreateDocument[TextCell[Column@cellIDCellTagsInfo,"Text"],
WindowSize->All,WindowMargins->{{Automatic,Automatic},{Automatic,10}}];

Tags With Data

Using tags to store data or expressions (but you can use TaggingRules as well.  E.g., CellTags->{"includes data", "MyData[{\"peaches\", \"apples", \"pears\"}]"}

TextCell["Perhaps there's some data here in the CellTags","Text",CellTags->{"IncludesData1","MyData[{\"peaches\", \"apples\",\"pears\"}]"},CellFrame->True,CellFrameLabels->{{"Data in CellTags",None},{None,None}}]//CellPrint

Perhaps there's some data here in the CellTags

Go to the cell and retrieve its data from its CellTags

Module[{tags,data,nb},
nb=EvaluationNotebook[];
NotebookFind[nb,"IncludesData1",All,CellTags,AutoScroll->False];
tags=CellTags/.Options[NotebookSelection[nb],CellTags];
data=First@Select[tags,StringMatchQ[#,"MyData*"]&];
Identity@@ToExpression[data]
]

Better yet to use TaggingRules

fruitData=Compress[{"Apples","Apricot","Avocado","Banana","Black Berry","Black Currant","Cherry","Clementine","Currant","Damson","Dates","Figs","Goose Berries","Grapes","Grape Fruit","Guava","Kiwi","Lemon","Lime","Mango","Mellon","Nectarines","Oranges","Peaches","Pears","Pineapple","Plumb","Prunes","Raisins","Raspberries","Rhubarb","Strawberries","Tangerines"}];

CellPrint[
Function[x,
TextCell["Perhaps there's some other data here in the Cell's TaggingRules","Text",CellTags->{"IncludesData2"},CellFrame->True,CellFrameLabels->{{"Data in TaggingRules",None},{None,None}},TaggingRules:>{"FruitData"->x}]][fruitData]
]

Go to the cell and retrieve its data from its TaggingRules

Perhaps there's some other data here in the Cell's TaggingRules

Module[{taggingRules,data,nb},
nb=EvaluationNotebook[];
NotebookFind[nb,"IncludesData2",All,CellTags,AutoScroll->False];
taggingRules=TaggingRules/.Options[NotebookSelection[nb],TaggingRules];
data="FruitData"/.Flatten[{taggingRules}];
Uncompress[data]
]

Trick: When there's an expression that you want to convert to a string, and that expression has strings within it, use Compress to preserve the interior strings.  ToString is not the right tool for this

FullForm@ToString[f["abc"]]

Oops, it's missing the quotation marks in the argument of f

temp=Compress[f["abc"]];
FullForm@temp

FullForm[Uncompress[temp]]

Make Your Own! Do It Yourself! Show No Fear!

Or get a copy of A WorkLife FrameWork...

See me to get a personal install of a trial copy, password, and tutorial...

Mathematica's Notebook interface is vastly underutilized

I encourage you to write your own tools that make use of tagging...