Search Pattern Cookbook

The Search function in TWiki is very powerful. Especially searches using a RegularExpression play an important part of tapping TWiki's full potential. Unfortunately RegularExpressions can be incredibly obscure to the uninitiated.

Most people not familiar (enough) with Regular Expressions mostly cut and paste (and maybe tweak) from existing examples. This page intends to collect lots of examples together.

Pattern 1: Extract values from a table

Problem definition

Suppose there is a topic with a table defining entries in a TWikiForm. I.e. they define select menu items in a form template. They are then formatted like:

| *Name* | *Type* | *Tooltip message* |
| option1 | option | |
| option2 | option | |
| option3 | option | |

How to extract the 'name' values, i.e. 'option1', 'option2' and 'option3' and put them in a HTML form select input?

Solution

The following search pattern can be employed:

<form>
<select>
%SEARCH{ "^\|[^\|]*\| *option *\|" topic="%TOPIC%" regex="on" multiple="on" nosearch="on" nototal="on" format="<option>$pattern(^\| *(.*?) *\|.*)</option>" }%
</select>
</form>

which is, in effect:

option1 option2 option3

Pattern 2: List generated from form classification

Problem

Imagine a TWiki form-based topic classification, i.e. every page has a form with several fields. How to:

1. create a search to display all topics where one form field is set to a certain value
2. create a search to filter the list above based on the values of a second form field

Test case

In practice:
Image a TWiki form with two fields:

• TopicClassification = One, Two or Three
• TopicStatus = Test or Final

We will:

1. List all topics where the TopicClassification field is set to 'Two'
2. Enable the user to filter this list based on the values of TopicStatus

Solution

%SEARCH{"[T]opicClassification.*value\=.*Two;[T]opicStatus.*value\=.*%URLPARAM{type}%"
regex="on" casesensitive="on" nosearch="on" 
format="   * $topic - <font face=\"arial,helvetica\" size=\"1\"> 
_last modified by_ $wikiusername _on_ $date </font> %BR% &nbsp;&nbsp;&nbsp; 
<font face=\"arial,helvetica\" size=\"1\"> $formfield(TopicStatus) </font>" 
sort="topic"}%

The filtering select dialogue is created as in Pattern 1:

%STARTSIDEBAR%
*Filter:* %BR%
<form name="selectType" action="%SCRIPTURLPATH{"view"}%/%WEB%/" >
<select name="type" size="1" onchange="document.location=this.value;"> 
%SEARCH{ "^\|[^\|]*\| *option *\|" topic="TopicClassification" web="%WEB%" regex="on" 
multiple="on" nosearch="on" nototal="on" format="<option value=%INCLUDINGTOPIC%?type=$pattern(^\| *(.*?) *\|.*)>$pattern(^\| *(.*?) *\|.*)</option>" }% 
<option value=%INCLUDINGTOPIC%>All pages</option> </select>
</form>
%STOPSIDEBAR% 

This will create similar functionality as TWiki:Plugins.TopicClassificationAddOn

Pattern 3: Creating lists of TWiki usernames

Problem

How to populate a list box with all usernames of registered TWiki users

Solution 1: Appropriate for Sep 2004 TWiki (Cairo)

<form name="testing" action="%SCRIPTURLPATH{"view"}%/%MAINWEB%" method="get">
<select name="topic">
<option>Select user...</option>
%SEARCH{ "Name:;Email:;Country:" web="%MAINWEB%" type="regex" nosearch="on" nototal="on" format="<option>$topic</option>" }%
</select>
<input type="submit" value="Go" />
</form>

Which expands to this: (here limited to all Z* users because TWiki.org has so many)

Select user... ZilmariaRocha

This searches all topics in the Main web that contain "Name", "Email" and "Country" bullets. Alternatively, do a FormattedSearch with multiple="on" on the Main.TWikiUsers topic.

Solution 2: As Solution 1, but with possibility for multi-selecting usernames

The example of Solution 1 produces the list box. Add a MULTIPLE to the select statement, i.e.:

<select name="topic" size="2" MULTIPLE>

Please note that the Search pattern is unchanged compared to Solution 1. The change is in the HTML form element.

The abovementioned modification is, in effect:

Select user... ZilmariaRocha

Solution 3: Appropriate for TWiki 4 (Dakar)

When the User information is stored in a UserForm (as is default in Dakar) then this list can be generated as follows:

<form name="testing" action="%SCRIPTURLPATH{"view"}%/%MAINWEB%" method="get">
<select name="topic">
<option>Select user...</option>
%SEARCH{"%META:FORM.*[U]serForm" web="%MAINWEB%" regex="on" casesensitive="on" nosearch="on" format="<option>$topic</option>" sort="topic" excludetopic="Test*, TWiki*"}%
</select>
<input type="submit" value="Go" />
</form>

In the above example:

• META:FORM.*[U]serForm will search for all topics with a UserForm attached - change this if you have a different form where userdata is stored. Please note that this search does not actually extract anything from the form - it just uses it to identify the appropriate pages
• excludetopic="Test*, TWiki*" allows to skip all topics starting with Test and TWiki, such as TestUser or TWikiAdmin. Use this if you have any special users who you do not want appearing in this list

Pattern 4: Extract the parent of a given topic

Problem

How to get to the parent of the current topic to display on the page?

Solution

You might think that the following Search would do the trick:

%SEARCH{ "^%BASETOPIC%$" scope="topic" nosearch="on" type="regex" nototal="on" format="[[$parent][parent_link]]" }%

However, the [[$parent][parent_link]] link fails if the topic has no parent set ($parent will be empty). You can use some SpreadSheetPlugin? magic to conditionally link to the parent or to WebHome: [[$percntCALC{$IF($EXACT($parent,), %HOMETOPIC%, $parent)}$percnt][parent_link]]

So the total Search query to find a topic's parent topic is:

%SEARCH{ "^%BASETOPIC%$" scope="topic" nosearch="on" type="regex" nototal="on" format="[<nop>[$percntCALC{$IF($EXACT($parent,), <nop>%HOMETOPIC%, $parent)}$percnt][parent_link]]" }%

Test Case

The parent topic of this topic is: FormattedSearch

Pattern 5: Search and display the home topics of public webs in a list

Problem

How to find and display public webs in a drop down list box.

Solution

Thanks to TWiki:Main.PeterThoeny for these solutions.

<form>
<select name="topic">
<option value="%TOPIC%">Select...</option>
%SEARCH{ "%HOMETOPIC%" scope="topic" web="all" topic="%HOMETOPIC%" format="<option value=\"$web.$topic\">$web</option>" separator=" " }%
</select>
<input type="submit"  value="Go" />
</form>

Test case

Public webs of TWiki.

Select... Search: WebHome

Abacos

Number of topics: 1

Acbahia

Number of topics: 1

AnpedGT16

Number of topics: 1

ArcoDigital

Number of topics: 1

Argumento

Number of topics: 1

Avsan

Number of topics: 1

CalculoB

Number of topics: 1

Ceb

Number of topics: 1

Cetad

Number of topics: 1

CetadObserva

Number of topics: 1

Cibercultura

Number of topics: 1

Ciberfem

Number of topics: 1

CiberParque

Number of topics: 1

ColoquioCiags

Number of topics: 1

Coloquiofasa

Number of topics: 1

ConexoesSaberes

Number of topics: 1

Cpdteste

Number of topics: 1

Cppd

Number of topics: 1

Creche

Number of topics: 1

Cridi

Number of topics: 1

Da

Number of topics: 1

DACN

Number of topics: 1

DCE

Number of topics: 1

DelzaTeste

Number of topics: 1

DeniseCarla

Number of topics: 1

DepHistoria

Number of topics: 1

DicionarioBelasartes

Number of topics: 1

Ecologia

Number of topics: 1

EDC

Number of topics: 1

Educandow

Number of topics: 1

EleicoesReitor2010

Number of topics: 1

Encima

Number of topics: 1

Enearte

Number of topics: 1

Estruturas

Number of topics: 1

EstruturasEng

Number of topics: 1

FACED

Number of topics: 1

FAT

Number of topics: 1

FepFaced

Number of topics: 1

GEC

Number of topics: 1

GeneticaBiodiversidade

Number of topics: 1

GeneticaBiodiversidade/GeneticaBiodiversidade

Number of topics: 1

Gepindi

Number of topics: 1

GetecEng

Number of topics: 1

Godofredofilho

Number of topics: 1

GrupoAlgebra

Number of topics: 1

ICI010

Number of topics: 1

Informev

Number of topics: 1

Ites

Number of topics: 1

LabioComp

Number of topics: 1

LEG

Number of topics: 1

Lepeja

Number of topics: 1

Letras

Number of topics: 1

LivroLivreSalvador

Number of topics: 1

Main

Number of topics: 1

MaisUm

Number of topics: 1

Mata07

Number of topics: 1

Mefes

Number of topics: 1

MefesCpd

Number of topics: 1

MetaReciclagem

Number of topics: 1

Neclif

Number of topics: 1

NelsonPretto

Number of topics: 1

Nuclear

Number of topics: 1

Numcad

Number of topics: 1

Nutricao

Number of topics: 1

Observa

Number of topics: 1

OrfaosdeRua

Number of topics: 1

PauloCostaLima

Number of topics: 1

PdI

Number of topics: 1

PescandoLetras

Number of topics: 1

PETFilosofia

Number of topics: 1

Pgif

Number of topics: 1

PGNUT

Number of topics: 1

PortalPpga

Number of topics: 1

PosCultura

Number of topics: 1

Pospetroigeo

Number of topics: 1

PPGAC

Number of topics: 1

PPGE

Number of topics: 1

PpggBio

Number of topics: 1

Ppggenbio

Number of topics: 1

Pretto

Number of topics: 1

Proad

Number of topics: 1

PROGESP

Number of topics: 1

ProjetoLencois

Number of topics: 1

Quimica

Number of topics: 1

RadioFACED

Number of topics: 1

RadioTeatro

Number of topics: 1

RadioWeb

Number of topics: 1

Riosymposium10

Number of topics: 1

Ripe

Number of topics: 1

Salasdoctai

Number of topics: 1

Sat

Number of topics: 1

Sedu

Number of topics: 1

SemBio

Number of topics: 1

SeminarioPibid

Number of topics: 1

SimoneLucena

Number of topics: 1

Sociologia

Number of topics: 1

SSL

Number of topics: 1

Tabuleiro

Number of topics: 1

TabuleirosUfba

Number of topics: 1

TCinema

Number of topics: 1

TerritoriosDigitais

Number of topics: 1

TWiki

Number of topics: 1

Twikidea

Number of topics: 1

UFBAIrece

Number of topics: 1

UniversidadeNova

Number of topics: 1

VizinhoEstrangeiro

Number of topics: 1

XIISNHCT

Number of topics: 1

For private webs, or any other webs you wish to exclude from the display, use "on" for the Exclude web from a web="all" search setting in the relevant web's WebPreferences topic.

Alternative solution

This result can also be accomplished with the %WEBLIST% variable.

Select... Abacos Acbahia AnpedGT16 ArcoDigital Argumento Avsan CalculoB Ceb Cetad CetadObserva Cibercultura Ciberfem CiberParque ColoquioCiags Coloquiofasa ConexoesSaberes Cpdteste Cppd Creche Cridi Da DACN DCE DelzaTeste DeniseCarla DepHistoria DicionarioBelasartes Ecologia EDC Educandow EduMus EleicoesReitor2010 Encima Enearte Estruturas EstruturasEng FACED FAT FepFaced GEC GeneticaBiodiversidade GeneticaBiodiversidade3 GeneticaBiodiversidade/GeneticaBiodiversidade Gepindi GetecEng Godofredofilho GrupoAlgebra ICI010 Informev Ites LabioComp LEG Lepeja Letras LivroLivreSalvador Main MaisUm Mata07 Mefes MefesCpd MetaReciclagem Neclif NelsonPretto Nuclear Numcad Nutricao Observa OrfaosdeRua PauloCostaLima PdI PescandoLetras PETFilosofia Pgif PGNUT PortalPpga PosCultura Pospetroigeo PPGAC PPGE PpggBio Ppggenbio Pretto Proad PROGESP ProjetoLencois Quimica RadioFACED RadioTeatro RadioWeb Riosymposium10 Ripe Salasdoctai Sat Sedu SemBio SeminarioPibid SimoneLucena Sociologia SSL Tabuleiro TabuleirosUfba TCinema TerritoriosDigitais TWiki Twikidea UFBAIrece UniversidadeNova VizinhoEstrangeiro XIISNHCT

Pattern 6: Extract a value from a bullet list

Problem

Display the user name in the user's topic title

Solution

Search for the Name: entry.

%SEARCH{"   * [N]ame: " topic="%TOPIC%" regex="on" casesensitive="on" nosummary="on" nosearch="on" noheader="on" nototal="on" format="---+!! $pattern(.*   \* Name: ([^\n]*).*)"}%

Test case

To create a test case, we will put a name entry here:

• Name: John Doe

Search result:

Pattern 7: Search for Form and Meta data: explained

Problem

Below is an example of a search that searches form data. The questions are:

• why is this searching the metadata, shouldn't it just search the text?
• what is the meaning of the td..td in the search expression?

%SEARCH{ "[S]tatus.*(td..td|value\=).*[W]aiting" casesensitive="on" regex="on" 
nosearch="on" nototal="on" format="| [[$topic]]<br /> ($date - $rev - 
[[%SCRIPTURLPATH{rdiff}%/$web/$topic][Diffs]]) |"}%

Solution

%SEARCH depends on grep, and grep searches the whole file, including the meta data.

An example meta data form field is:

%META:FIELD{name="OperatingSystem" title="OperatingSystem" value="OsWin"}%

So a search for a form field could look like:

%SEARCH{ "[O]peratingSystem.*value\=.*[O]sWin" regex="on" ... }%

• Using square brackets is a trick to avoid a hit on the topic doing the search.
• The .* indicate that there can be any number of any character between OperatingSystem and value in the (whole) file

Now the original file format of the category table (the predecessor of the TWiki forms) looks like this:

<td valign="top" align="right"> OperatingSystem:  </td><td>  OsWin </td>

The following search finds topics in the old and new format:

%SEARCH{ "[O]peratingSystem.*(td..td|value\=).*[O]sWin" regex="on" ... }%

The td..td matches td<>td; a simple search on "[O]peratingSystem.*[O]sWin" could find a hit in the topic text by coincidence.

A simple %SEARCH{ "[O]peratingSystem.*value\=.*[O]sWin" ...}% search is sufficient if you do not have topics in the old format.

Pattern 8: Search all topics that have been moved

Problem

How would I go about listing all moved topics ?

Solution

Search for the META:TOPICMOVED meta data. Type this:

Moved topics: %SEARCH{ "%META\:TOPICMOVED" regex="on" format="$topic, " nosearch="on" noheader="on" nosummary="on" }%

to get this (limited to 10 results):

Moved topics: FreeSkin,

Contributors

TWiki:Main.AntonAylward, TWiki:Main.ArthurClemens, TWiki:Main.JosMaccabiani, TWiki:Main.PeterThoeny, TWiki:Main.SueLocke

Related Topics: UserDocumentationCategory