Text formats and how to handle them

Silk purses and sow's ears

• Can you make a silk purse out of a sow's ear?
  • (or, in French, faire d'une buse un epervier)?
  • (or, in Italian, la botte dà il vino che ha non si cava sangue da una rapa)
  • or, maybe, in Russian, из говна конфетку не сделаешь
• Documents come in many different formats, but these formats are usually intended to represent a common set of features.
• Given the right tools and a common set of features, we should be able to convert (transduce) from one format representing a given set of features to another representation using a different format.

Up to a point...

Semantic vs presentational features

• Some formats aim to capture semantic features of a document ("heading", "paragraph", "title")
• Others aim to specify the way a document should be formatted ("font size", "justification", "italic"...)
• and in many/most the distinction is not clear : for example, what does <p> mean in HTML? or <br> ?

We are where we are

Documents might come in...

• so-called PVA -- plain text with no encoding
• plain text with esoteric codes
• Word or ODT : word processor formats
• (X)HTML or eBook : web formats
• Typesetting formats (inDesign, TeX...)
• XML or TEI-XML
• PDF

Low hanging fruit...

• A PDF file preserves only the appearances of an original: you can (painfully sometimes) extract letters from it by clever programming, but it's usually quicker to just print it out, and put it through an OCR process
• A TEI-XML document should come with an ODD defining its use of the TEI vocabulary: use that to help you write an XSLT stylesheet to extract just the elements and attributes you want
• Plain text : as you've seen, it may be quite easy to add minimal TEI tagging, provided that the text is consistent in e.g. its use of linebreaks, whitespace, capitals, and other things that your regular expressions will be relying on

Word or Libre Office vel sim

• These are in fact XML formats respecting (more or less) the OOXML standard ... up to a point
• A Word file is really a compressed archive, like a zip, containing numerous document fragments
• oXygen can unpack and transform the useful part into TEI for you
• Word styles are preserved : so if they were used semantically in the original, converting them to semantic tagging is easy
• But there are always surprises...

XHTML and friends

• HTML looks superficially like XML, but is not necessarily well-formed and rarely validated
• XHTML is an XML-conformant expression of HTML: use the invaluable tidy utility to generate it
• eBooks, like docx, are really compressed archives containing many different XHTML or other files
• The organization of an ebook is more or less well-standardized but because the focus of the encoding is snazzy presentation, there is much variation, for example in style names and behaviours

Typesetting and other esoteric formats

• You may encounter legacy type-setting formats such as TeX or inDesign
• These formats represent the desired formatting of the text, but not (usually) its semantics
• Conversion from XML to these formats is useful and fairly straightforward (there is also a standard called DSSSL of relevance here); the reverse is not the case

Good luck!

Two essential technologies: regexp and xpath

Regexp (regular expressions) allow you to specify patterns which match strings of characters and manipulate the resulting matches

• xyz : the string xyz
• [xyz] any one of the characters x y z
• [\d+] one or more digits
• [\p{Lu}+] one or more uppercase unicode characters

Xpath is a standard syntax for matching parts of an XML tree in terms of its elements and attributes

• //p : any <p> anywhere
• //p[anchor] : any <p> containing an <anchor>
• //p[@rend] : any p with a rend attribute
• //p[starts-with(@rend,'Head')] any p whose rend attribute has a value starting with Head

Regexp and xpath are both built-in to oXygen

Programming options

If you have lots of texts in a specific arcane format, it is worth investing time and effort to translate them, using whatever tools are at your disposal.

A multi-stage path is usually easiest, e.g.

• use tidy, or perl or python or whatever language you like to convert to well-formed xml
• use XSLT to convert arbitrary XML to TEI ELTeC

Remember: the computer should be doing the boring repetitive work, not you!