Lightweight Markup

If you've browsed the source files for this blog or for the forthcoming cyborg institute wiki, ciwiki you may notice that my work is mostly stored in "markdown format" which is a lightweight markup language for generating richly formatted text. Markdown isn't the only format around, indeed formats like reStructured text and textile are other very similar tools. Different, but they achieve same purpose. What purpose, you ask anonymous inquirer? They make it easy and painless to write text with links, emphasis (italics), structure (headings), and other features, in an easy to read plain text format.

The truth is that there isn't a really good way for "rich text" (bold, italics, links) to be stored in a format that's both universal enough for computers to read in multiple context (desktop applications, on the web) that's also easy for humans to read and write in a consistent way. Sure there's (x)HTML which is a great format for computers to read, but its hard to write and difficult to read casually. At the same time word-processor formats (.rtf, .doc, .odt, etc) render inconsistently across platforms and don't work as well for use on the web, or for email.

Lightweight markup formats, and particularly markdown, provides a solution to this problem. Rather than creating a format that will render well on web-pages, on paper, and in plain text, these formats provide a limited syntax for most common markup needs and then provide a script to translate this "lightweight" language into other formats. These scripts exist as plugins/modes for most common text editing software, and are implemented in a great many programming languages so that standard lightweight markup can be used in many contexts.

Here's the rundown (note, I'll use markdown as the example but these facts all hold true for other similar products):

• Markdown is human readable, so in some cases (like email, and collaborating with other writers) there's no need to convert a file to XHTML or some other format.

• Markdown is designed to replicate many common editing conventions for plain-text email. So the chances are, that you already know much of the syntax and can understand texts written in markdown quite readily.

• Markdown generates standards compliant XHTML. Even sloppy markdown generates compliant XHTML. XHTML is easy to translate into other formats (including word processor formats), and non-trival to generate perfectly "by hand."

• Markdown interpreters are written in many languages, including Perl, PHP, Python, C, Lisp, Java, C# and so forth, which makes it particularly easy to integrate markdown into whatever environment you're used to working in.

• There are tools, like Maruku (and others) that convert text to other formats (like PDF), for non-web output.

• Text editors (including, vim, emacs, textmate, notepad++, bbedit) have support for syntax highlighting for markdown, so that your editing environment provides a rich interactive environment while still editing simple plain-text files.

• Markdown makes it easy to "live in plain text" without sacrificing features that we, as writers of words, are accustomed to.

The benefits of using markdown, are perhaps most fully realized in the context of using plain text itself, but that's another argument for another time. And as always, if something in this post struck a chord with you, but you don't quite know how to integrate it into your workflow directly, that's something we can work on together.

tycho's notebook archives

information-repository-systems
Posted Thu May 6 21:20:23 2010

archive
Posted Sun Sep 27 17:27:22 2009

Journal Keeping and Empiricism

In talking about the way I approach computer usage I found my self saying "you know it's all very empirical."

And I was immediately struck by both how true this, and how awkward it made me feel. As a former (recovering?) student of Women's Studies and a not-particularly-quantitative social scientist, I'm far too used to "empiricism" being deployed as the [ahem] epistemological whipping stick of enthroned ideologies/groups/etc.

But it's also true: empiricism is all about observation, and data collection, and analysis. It's about recording processes, and outcomes, and it's about gathering information--data--for later analysis. Not evil. And particularly as I work on figuring out how to help people use computers and technology more effectively, empiricism is something I can totally get behind. In fact, I think it's totally essential.

Now, from the title, surely you can see where this is going.

While I think there are aspects of our interactions and use of computers that we're not particularly good at keeping track of and analyzing (e.g. usability testing with regards to eye-tracking), there are some really crucial higher level areas where we are capable of understanding in ourselves: how we organize files, how we organize notes, fact files, how decide to customize software, how we output/publish/share our data, and so forth.

These sorts of concerns are relevant to all computer users and transcend the big debates, like command line vs. gui; modal vs. context aware, and the small debates, vi vs. emacs; windows vs. mac; open source vs. proprietary; markdown vs. textile.

It's "be universal" day on tychoish, apparently.

In any case, I think keeping a journal of a couple of key things can help us all figure out how to do things better. It's empiricism, on a small scale. The system you use to keep track of things doesn't matter too much: just as long as its easy for you to reflexively keep track of a few details every day and then review all this data at a later point. I'm partial to just keeping one journal, rather than project/context-specific systems, but that's personal presence more than anything.

I should say that I've started keeping a journal myself but I've admittedly not been particularly regular/rigorous about it. For my benefit and yours here's the list of things that I think would be particularly important to track:

• Word Count, or some other numerical measure of progress that's relevant to whatever your core practice is. Words make sense for prose writers, line counts for programmers, minutes of practice for musicians, etc. Whatever makes sense, and hopefully something fairly concrete.

• Contextual Variables: things affect our productivity and work in our lives. If there's something that happened that might make your/my word count unusually high or low, I imagine it'll be helpful to keep track of this. Numbers are helpful--particularly in the aggregate--but if you're reading through your journal day by day, and you have a day where you wrote much less than you did most other adjacent days, being able to say "ah, I had meetings/workshops all day that day, no wonder I didn't get anything done."

• Changes to your "system:" If you change things about the way you work, it's good to note when this occurs, particularly in retrospect so that you can evaluate the utility and efficacy of changes.

• Discoveries/Important thoughts. I suppose many people might use their public blogs for this forum, but I tend to write about a week out, and I often write posts out of order, so having a file that lets me track what I'm reading, and what the big issues on my mind are, is really helpful.

I'm sure you all have some journaling tips. I've been using the org-journal bits that jack wrote wrote, though I'm thinking that I might need to add a few snippets to take care of org-mode tags so that I can filter for some data. I know there's a similar bundle for ?TextMate that I used back in the day. And of course, this is nothing that wouldn't work beautifully in a private LiveJournal account, or private instance of WordPress. The details don't matter much, but if any of you keep journals for this kind of thing (or track your computer usage productivity) and have other better ideas, I'd love to hear how you all deal with this.

Onward and Upward!

git, flashbake, and audience

I've mentioned a while back (surely) a program called flashbake, which Gina Tripani covered on lifehacker a while ago, and while I don't think I could add anything this review, I can provide you with links, and talk a bit about how hacker-types get their starts, and how audiences for free software are formed.

This is, I think, a big issue facing the free software movement, basically: we build great software, how do we get (more) people to use it.

The issue is that free software doesn't have advertising budgets, free software projects don't have muscle with operating system vendors (MS) to bundle applications, and for the most part free software projects can't get their software distributed on hardware (eg. the dell-Microsoft relationship).

And then on top of requiring users to actively choose open source, a lot of free software requires a certain level of technical knowledge, and the documentation is poor, and open source projects can't (generally) afford the kind of usability studies that closed-development projects rely on, and as a result it often takes a few generations for applications to really work.

The conventional response is to try and make software as usable and intuitive as possible, and I've written before about the problems of treating your users like idiots and some issues regarding user experience issues, and in general I think the conclusion is: our goal should be to unobtrusively turn less technical users into more technical users, rather than turn more technical computing domains into less technical computing domains. Make people smarter, rather than make computer interfaces smarter.

I think, in a weird way, GNU Emacs does this pretty well. A hacker might not know very much (emacs-)Lisp when they start to use emacs, but for the most part it doesn't take people very long to get to a point where they can hack things together in elisp and understand what's going on in the emacs code that they download.

So back to git and flashbake. Right.

I actually don't think that flashbake treats users like idiots. In fact, git was built so that new interfaces could be added to the system without breaking compatibility with other git repositories.

Git is a filesystem, basically. A microfilesystem, if you will, with a notion of versioning--storing iterations across time--and distribution--replicating the store across locations. In order to make both of these features work, it has to have top level support for branches and merging (and it does).

But beyond those capabilities, what goes in to git doesn't seem to matter much. And once you start using git, and you get a feel for how it works, the more you want to put into it.

Flashbake is just a tool for putting writing into git.

Programmers who aren't familiar with how writers work and think may say "why would you want to commit so often?" Or, "why would I want to know what the weather is every fifteen minutes while I do my work?" But I think it makes sense, and makes it possible a sort of digital archeology of a text that writers--myself included--are really intrigued with. Particularly given that "the cost" of running a script like this in the background is so small.

Anyway, give it a try if you're so enclined.

Thoughts on Collaboration

The Internet is an amazing tool for information procurement, and increasingly more of our culture and knowledge is delivered using Internet technologies: this is pretty easy to grasp. Knowing this, it's startling to think that the Internet as information delivery technology is largely overshadowed by the Internet as a technology that facilitates collaboration.

We're used to technologies like the printing press, radio, television, the telephone, the xerox machine, etc. vastly expanding the facilitation of dissemination and spread of information and knowledge, so it's easiest to think of the Internet in these terms. At the same time what makes the Internet so powerful is that it bridges the distance between people and provides technologies that in making it easier to share information, groups of people are able to create information on the Internet.

Open Source and Free Software is a result of this collaboration, for example, but the Internet is full of examples of communities coming together to produce information and resources that are bigger than any one person could create on their own: social networking sites (ravelry.com, facebook, nings) blogging communities (eg. livejournal, wordpress.com) wiki-projects (eg. wikipedia, flossmanuals c2-wiki.) In truth the technologies of the Internet that mediate these projects are relatively simple. For example:

• Mailing Lists

  Discussion lists are ancient in the land of the internet, but many communities function very well with just a mailing list. We are often overwhelmed by email, but there are many features which recommend email over newer tools like blogging. Primarily, email is "pushed" to your members'. Since we generally presume that people check their email, getting information in email requires less intention than getting information via blog or wiki, while audience can remain stable for longer.

• Wikis

  Wikis are a great tool for collaborating on a text which is uniquely digital, uniquely hypertext. While encylcopedists have adopted it to great effect, the form is flexible, discursive, and highly interactive. As a result it can be--with proper guidance--a great tool for collaborative projects, but unguided they often are too flexible and underutilized as a result.

• Blogs

  The blog is perhaps the easiest technology to use, and use well. We're familar with the form if not from other blogs, but from columns and analog journals. The biggest challenges are keeping blog posts indexed and useful for more than a few weeks, and learning to blog as a habit. Blogs make a great experimental space for playing with new ideas, in addition to recording personal perspectives and historical contexts.

• Messaging

  Before instant messaging, IRC (internet rely chat) provides group chat experiences that allows collaborators to "talk" things over in real time, in groups, both as a primary form of communication and as a "back-channel" for distributing links and other information in other forms of real time conversations. Often, however, such messaging can be distracting, and hard to follow for the uninitiated. Furthermore there are some kinds of projects for which sporadic telephone calls are more productive.

• Social Networking

  While "work" in the conventional sense isn't often accomplished on social networking sites, they do foster community and communication, and contemporary functionality includes a "feed" of information that can automatically keep your team in touch with their collective activities.

• Version Control

  Programmers use version control systems (subversion, git, etc.) to allow a group of people to work on one project concurrently, to store chronological iterations of their projects so that they can return to "known working states" if an train of thought leads to a dead end. The technology is quite simple, but remarkably powerful in the creation of shared works, while still allowing and respecting individual contributions. Software engineers use these tools to great effect, but I'd argue that additional kinds of creators and creative teams should use version control and similar tools.

This isn't to say that all collaborative tools are perfect and we don't need new and more inventive tool to facilitate collaborative work. I've touched on some of the more specific challenges, but collaboration on the Internet--as a whole--face one major challenge: one of expectation.

When we understand how simple technologies are, when we see the great accomplishments of Internet technologies it's all to easy to say "well wouldn't it be great if I did that," or "you know, why [my project] really needs is some good collaborative technology." It turns out, not surprisingly, that harnessing the power of collaborative technology this is much more difficult than simply flipping a few bits on a piece of software and then waiting for emergent phenomena to develop.

This challenge, how to facilitate and shepperd a nascent community--even when that community is all located in one place--can take many different forms. While the specifics of these challenges form the basis of much of our work here, and are highly dependent on the goals, resources, and contexts of individual projects there are some general themes:

• Communities need editors and moderators to provide leadership and ensure quality.

• Raw information is rarely useful without curated guides and views onto the data.

• Communities need the flexibility to have wide reaching discussions and sometimes stray from topic, in order to develop unique identities.

• Beyond an initial set of necessary tools, community needs should dictate the development of technology, and often fewer tools and platforms are better than more tools and platforms.

On Customization and Adaptation

One of the applied issues related to cyborg interactions that I find myself contemplating with some (too much?) frequency is the problem of customizing your computing interfaces. Problem? Well here are two arguments that I've been considering:

1. Customization of our computers and our computing interface allows us to work more efficiently and naturally, so that the computer just works the way we need it to and doesn't get in our way. When I say customization, I mean, tweaking the configurations on your text editor/editing program, creating shortcuts, using programs like Quicksilver emacs, or awesome to shape your environment.

   No matter how geeky you are, though, we all customize our environment somewhat. Some of us are a bit more into this than others, of course, and I'm a big fan of customization, obviously (or not) but customization isn't without it's drawbacks...

2. Which are, that customization means additional learning curve, and increased start-up costs whenever you get a new computer. Additional learning curve because you have to learn what a program/system is capable of to begin with, what needs to be changed in order for you to work better, and then you'd need to know how to change it. Which takes some know how.

   Increased start up costs are a bit more complex. Basically if you customize your system/applications, then when you get dropped into a system that's new or with which you're unfamiliar, you have to spend some time either adjusting back to the defaults or customizing things to work with your system. The even worse corollary to this is that you use multiple system you also have to keep the customizations on all of your systems in sync, other wise, it's all sorts of complicated. There are ways to ameliorate these problems, but they have to be considered.

I, obviously, fall into the "customization is optimal" school of thought, but the other school--in many circumstances--has a lot of merit for some users. This is one of the topics that I'll be exploring more in these blog posts, that we can discuss on the wiki and/or that you and I might discuss in a more applied context if you or your organization would like to discuss your work and technology with me. In any case, I look forward to hearing from you!

About tycho garen

Speaking of my byline, I should make clear the tycho garen/Sam Kleinman distinction. It's a long story, the origins and reasons of which aren't particularly interesting or relevant, but some years ago, I began to feel uncomfortable with the lack of privacy, and potential ?confused contexts that goes along with contemporary culture. So I began writing my blog as tycho garen, when I publish fiction I use it as a byline. I send emails (by default) from tycho, I was photographed for a portrait project, and put tycho garen on the model release (as the name). tycho (never capitalized, it seems) is even the originator of my PGP key.

I wouldn't want to make things as cut and dry as saying that "Sam" is the ?meatspace identity and tycho is the cyborg/cyberspace identity, though properly contextualizing my work and relationships is one of the inspirations for using "tycho." In the beginning, I suspected that the domains Sam would work in and the domains that tycho would operate in would be distinct enough to avoid overlap or collision, but time proves me wrong.

If you wondering about the method to the madness regarding how I choose which moniker to use, here's my intention: Sam will cover all of the public-facing and administrative content for the Institute, but on the wiki I will sign contributions (when appropriate) with tychoish and a link to this page. Because it only seems fair.