Last December Amazon enabled its Kindle Personal Documents Service for iPhone/iPad users, assigning each device a new email address, and this month the same feature has been enabled for Android users. Our Kindle It service has up to now been able to send to only one Kindle email address at a time, but as of today you can enter up to 5 addresses (separated by commas):

This will also work with our Push to Kindle Android app (no update necessary).

Let us know if you have any trouble.

To work for a corporation is to be part of a system in which power flows strictly from the top down – it’s a totalitarian power structure. That’s the lot of enormous numbers of people in the world. If you try to opt out in the UK, you are ordered to ‘look for work’, mostly different kinds of corporate bondage.

…

Our freedom is undermined by less obviously political forms of social manipulation. For example, we’re propagandised by society to seek freedom in dependency. One of the key purposes of modern schooling is to instill a thirst for ambition and status in the young. The emphasis is on competition, coming first, getting the best grades to get to the best universities to get the best jobs and salaries. This version of freedom chains us to external sources of reward and respect. If we believe we need high status and ‘success’ to be happy, we are chained to the people and organisations with the power to bestow these rewards. So we are trained to actually seek, to willingly embrace, a life of dependence.

Read more: Free to be Human: An Interview with David Edwards

Also well worth a read: Falling by David Edwards

What’s new?

Edit titles

If an article doesn’t have a title, or if we misidentify the title, you can now edit it before sending to the Kindle. Simply click the title block in the preview box and you’ll be able to change it.

Improved block quote support

If an article uses block quotations, these should show up indented now on the Kindle.

Clickable source URL

The URL of the article which we append to the end of the piece is now clickable.

Report problems

We know that some articles do not work well with Kindle It, if you encounter one of these, feel free to use the ‘report problem’ link to let us know. The URL will be prefilled and you can report the problem anonymously. If you want to know when it’s fixed, you can supply your email address and we’ll let you know.

New design

We’ve redesigned the interface using the Bootstrap toolkit from Twitter. We hope you like it.

@kindle.com sending enabled for sustainers

Previously we only sent articles to @free.kindle.com addresses – Amazon delivers these to Kindle 3 owners via Wi-Fi, and it’s free. Using the @kindle.com address allows you to receive articles via 3G if Wi-Fi is unavailable, although Amazon will charge for 3G delivery.

For older Kindle models which don’t have Wi-Fi, the @kindle.com address is the only way to get automatic delivery (downloading the MOBI and transferring it via USB is the other option, and doesn’t cost anything). Because @kindle.com sending can cost users, we’ve restricted it to sustainer accounts…

Sustainer accounts

We’re now inviting users who like the service to consider signing up for a sustainer account for 5 Euros. Sustainer accounts help us to maintain the service and continue developing it. More information.

Regarding the last point, I want to stress that the service will remain free. Everything which worked before, should still work the same way – we’ve improved things but haven’t disabled any features. Sustainer accounts are primarily a way for users to help us cover costs and sustain the project. Sustainers do get a few benefits – if you’re interested, there’s a 30 day trial to test the @kindle.com delivery.

Extensions

The most convenient way to use Kindle It is by using our browser extensions. So far we have extensions for Chrome, Firefox and Android. If you use a different browser, we also have a bookmarklet.

Feedback

We’d love to hear what you think. We’ve had some nice comments on Twitter so far, including:

“I think I’m in love: fivefilters.org/kindle-it/ – it’s perfect.” (Translated from Portuguese) — @myriamkazue

“Kindle It sends web pages to your Kindle and it’s insanely good.” — @coldclimate

“The second most useful extension (after firebug) in Firefox” (Translated from Spanish) — @sourcerebels

“@Evenwing @fivefilters ooh…I like this..” — @Yappings

“2 things = joy. 1: if you have a Kindle you can push long web articles to it with Kindle It http://bit.ly/ozPyWb 2: Tor http://bit.ly/HcV9F” — @kjftec

“@JayGreasley thanks for the tip on fivefilters.org, through it I found @medialens and that’s my type of news!” — @tashacres

If you have any trouble using it, email us: fivefilters@fivefilters.org

See the video below to see how the Android app works:

Try it: Kindle It

If anyone’s interested in the paper I submitted, it’s copied below.

Abstract

This paper looks at the ways in which professional interaction designers, despite the all too common rhetoric about serving humanity, end up uncritically serving corporate needs. It covers the conflict between the priorities of business and the goal of design; the influence of universities in setting students up to serve business interests; and how designers can resist by pursuing their own goals as radical professionals.

Introduction

[...] we ignore the fact that we too bow down to power, [...] to the anonymous power of the market, of success, of public opinion, of “common sense” – or rather, of common nonsense – and of the machine whose servants we have become. (Fromm, 2003)

Many professional interaction designers today have convinced themselves that their work is about serving humanity and meeting people’s needs. What is less often discussed is whether the goal of big business – who most professional designers work for as employees, or depend on as their clients – is actually compatible with a humanistic vision commonly associated with design. This paper looks at how the main beneficiaries of design work and design education are primarily self-interested corporations, and how work for them does not always translate into a good deal for humanity.

In the 1960s, the designer Victor Papanek wrote Design for the Real World (Papanek, 1985) to highlight the ‘social context of design’. He was concerned that designers had lost sight of what was important in design: its altruistic goals and potential for contributing to positive social change. The focus on superficial things, such as flashy car exteriors, gimmicky products and advertising, showed that designers had become uncritical servants of business and the design world had been blinded to real world problems. In the book, Papanek does not shy away from holding ‘tame, captive designers’ responsible for this state of affairs.

The aim of this paper is to try and explore how and why designers are still to a large extent serving corporate needs and often busying themselves with superficial things. The types of design projects that receive attention may have changed since Papanek’s time, but the greed-driven agendas have not.

Working for Whom?

Much of what designers do relies on the sanction of corporations and support from industry. In many schools, student projects are carried out in close cooperation with business, thus exposing students early on to the ‘real world’ of business constraints and priorities. While students are learning to navigate these constraints and come to grips with the priorities, are student attitudes and values also changing to reflect those of their enablers?

A fascinating study of economics students found that the further they progressed in their education and the more they were exposed to the self-interest model of economics, the more self-interested they themselves became – exhibiting more selfish behaviour compared to students studying non-economic subjects, and even compared with other economics students in earlier stages of their studies who hadn’t yet had as much exposure (Frank et al., 1993). Could the constant exposure to business priorities and goals be affecting interaction designers in a similar way?

David Noble, historian of technology and former professor at MIT, has written about his own experience teaching and working with technical people:

[...] it is the support of those in power (in our society, those with money or those with political, military, or legal authority) that affords technical people the luxury to dream, to dream expansively (yet within well-understood limits) and to make their dreams come true (by imposing them on others). Although most scientists and engineers would admit to their dependence upon those with power, few would concede that this relationship actually influences the way they think about things. [...] judging from my own experience [...], I know that few engineers are deliberately out to destroy jobs or unions or to harm people in any way. Although, of course, in practice they must satisfy the requirements of their bosses, their clients, or their customers, ultimately they aim only to do the best work for the good of society. Yet consistently, again and again, they turn out solutions that are good for the people in power [...] but often disastrous for the rest of us [...] (Noble, 1995)

I think this is an apt description of the situation designers are in. They too are dependent very much on ‘those in power’, particularly large corporations. If a design does not fit with a business plan, it is not considered useful, even if it may be of great benefit to society. This view is so ingrained in many designers’ minds that it prevents alternatives from being explored. As Anthony Dunne and Fiona Raby point out, ‘[d]eveloping a critical perspective in design is made difficult by the fact that the design profession, and product designers in particular, see the social value of their work as inextricably linked to the marketplace’ (Dunne and Raby, 2001).

I have found through various conversations with design students that many tend to view business in a positive light. Polls of the general public suggest a very different view, as noted by Dan Hind (2010): In 1996 a Harris poll found that 71 per cent of Americans thought that business had too much power over people’s lives; In 2005 another poll found that 90 per cent of Americans thought big business was a threat to democracy; Similarly in the UK, a 2006 report by the Joseph Rowntree Charitable Trust and the Joseph Rowntree Reform Trust noted that ‘research and evidence displays that many people feel business has too great an influence over government at the expense of the “ordinary” citizen’ (Hind, 2010). It’s worth noting that these are far more significant concerns than whether companies are producing well-designed products or services; they point to a real worry about the power, influence and behaviour of corporations and their effect on society.

Corporation as Psychopath

Much of the problem lies with the design of the corporation and the behaviour this produces. The lawyer Joel Bakan explains: ‘The corporation’s legally defined mandate is to pursue, relentlessly and without exception, its own self-interest, regardless of the often harmful consequences it might cause to others’ (Bakan, 2005). Robert Hare, a psychology professor, has shown that there are strong parallels between the behaviour of corporations and that of psychopaths. Many of the symptoms used to diagnose patients as psychopathic are symptoms exhibited by major corporations today: ‘callous disregard for the feelings of other people, the incapacity to maintain human relationships, reckless disregard for the safety of others, deceitfulness (continual lying to deceive for profit), the incapacity to experience guilt, and the failure to conform to social norms and respect for the law’ (The Corporation, 2003).

This reality is very rarely discussed. I think there are a few reasons for this. One is designers’ understanding that they are dependent on large companies for much of the work they do, which makes them reluctant to bite the hand that feeds. Another is that universities which train professional designers are increasingly depending on large companies for funding. A recent study looking at corporate influence in universities found that:

Companies have expanded the number and range of partnerships with universities, focusing on business research priorities and goals. The power and influence of some corporations, and the increased [...] pressure on researchers to bring in funding from business, means that academic departments are increasingly orientating themselves to commercial needs rather than to broader public interest or curiosity-driven goals. (Langley and Parkinson, 2009)

In addition to this, there is the issue of our general attitude toward self-interest (the guiding principle of corporations). The humanistic philosopher and psychoanalyst Erich Fromm noted the increasing tolerance towards self-interest and identified this changing attitude as a ‘socially patterned defect’: a defect which we share with so many others that we fail to see it as a defect at all. (Fromm, 2003) Quoting the 17th century philosopher Spinoza, Fromm writes:

For Spinoza, mental health is, in the last analysis, a manifestation of right living; mental illness, a symptom of the failure to live according to the requirements of human nature. ‘But if the greedy person thinks only of money and possessions, the ambitious one only of fame, one does not think of them as being insane, but only as annoying; generally one has contempt for them. But factually, greediness, ambition, and so forth are forms of insanity, although usually one does not think of them as ‘illness’”(Ethics, 4, prop. 44). In this statement, so foreign to the thinking of our time, Spinoza considers passions that do not correspond to the needs of human nature as pathological; in fact, he goes so far as to call them a form of insanity. (Fromm, 1997)

Fromm sums up:

In contrast to today’s belief that persons driven mainly by greed for money, possession, or fame are normal and well adjusted, they are considered by Spinoza utterly passive and basically sick. (Fromm, 1997)

Naturally, if we no longer see self-interest as a form of insanity, if we no longer even find it annoying, then we’re unlikely to feel compelled to change or resist corporations that are driven by it.

Serving Corporate Needs

Businesses, aware that their practices are motivated by self-interest, have for many years worked to create a benevolent public image of themselves. This image reinforces the feeling designers have about their own work being meaningful. To keep the illusion alive, companies will highlight examples of their products and services being put to good use to save lives, help people in need, or simply make ordinary people’s lives easier. Consider Google CEO Eric Schmidt’s talk at the 2010 Mobile World Congress. He gave the talk shortly after the Haiti earthquake. In his introduction he said:

Maybe the best testament to what you all, we all, have done is the way in which we’ve affected ordinary people’s lives. And there’s no better test of that than somebody in trouble. A famous story from a few weeks ago that somebody used an iPhone application to diagnose his wounds sitting in the rubble in Haiti. Or a teenage girl who updated her Facebook status sitting in the rubble to send rescuers to come rescue her. It must make you proud to have built something so fundamental to human existence, if you work in those companies or you’re part of this ecosystem that made all that possible. (Eric Schmidt at Mobile World Congress, 2010)

There is no denying that technology can help save lives and serve humanity, but these examples of Facebook and mobile technologies being used to save lives are incidental. The corporations behind these developments are driven by very different motivations. The good use these products are put to are highlighted to justify the work designers do, while at the same time masking the entire greed-driven enterprise and portraying it instead as a great beneficent endeavour.

In the three sections below I will look at a few ways in which business concerns influence what designers do.

Dark patterns

One way professional designers serve corporate needs is in devising and contributing to many questionable designs in their day-to-day work. The designer Harry Brignull started the Dark Patterns (2011) project to identify and expose designs which work against the public interest.

What’s interesting about the project is that its focus is not on the more obvious and small-time tricks and scams that many of us are exposed to through, for example, spam emails. Instead it puts the spotlight on the practices of large and well-established corporations who have put their considerable resources into implementing these self-serving tactics.

One of the patterns listed on the site is ‘Privacy Zuckering’, named after Facebook CEO Mark Zuckerberg. It is defined as ‘The act of creating deliberately confusing jargon and user-interfaces which trick your users into sharing more info about themselves than they really want to’ (Dark Patterns, 2011). Companies such as Facebook benefit when people share more information and more widely because, as security expert Bruce Schneier explains, they

[...] make their money by controlling vast amounts of their users’ information. Whether through targeted advertising, cross-selling or simply convincing their users to spend more time on their site and sign up their friends, more information shared in more ways, more publicly means more profits. This means these companies are motivated to continually ratchet down the privacy of their services, while at the same time pronouncing privacy erosions as inevitable and giving users the illusion of control. (Schneier, 2010)

Facebook ratchets down the privacy of its service by using very lax default privacy settings for users. These default settings have changed over the years from offering users more privacy to making user data more open to everyone (McKeon, 2010). Users can change the default privacy settings, but to do so they have to go through multiple, confusing privacy setting pages.

A much improved and easier to use interface has been proposed by designer James Home. This places a simple slider on the user’s home page which acts as a reminder of the user’s current privacy level (ranging from ‘Private’ on the left and ‘Open’ on the right). Users can also use it to change their privacy settings by simply dragging the slider right there on the home page. This is clearly a big improvement on the current design. However, as noted on the Dark Patterns website, it is unlikely to be adopted by Facebook. The reason comes down to the priorities of Facebook as a business. Giving users easy access to their privacy settings through a simplified interface could lead to more users opting to make their information more private – a move which would make Facebook less desirable to its partners and advertisers, and ultimately affects its profits.

It is worth noting that the designers responsible for such designs are not incompetent – companies the size of Facebook employ very talented, professional designers. Yet, as the example above and the many more listed on the Dark Patterns website show, faced with doing what’s best for ordinary people or serving the business’ interests, professional designers, consciously or not, will tend to do what’s best for the business.

Human-Centred Design

Human-centred design (HCD) is popular in the business world not because big business cares about humanity, but because understanding human behaviour is key to selling things or services to people. Understanding human behaviour is thus an essential component of interaction design. But the term human-centred design can give the misleading impression that those who practice it care about human beings in their totality. While this may be true of some practitioners, it is not the purpose of HCD. HCD concerns itself with human beings on a fairly shallow level, the focus being on satisfying the individual as primarily a consumer and potential customer.

Explaining the popularity of HCD, Bezerra and Brasell-Jones (2005) write that ‘the reasons this approach has become a trend has more to do with the economical benefit it brings to the companies who adopt it, than due to any ethical values.’ My own experience backs this up: in the first session of a HCD course I attended, our teacher, eager to impress us with job prospects, told us that one of the most successful students from a previous year now works for a large company designing the controls and interface used by fighter pilots in military aircrafts. Needless to say, for many of us the idea that human-centred design could be applied to things designed to harm humans seemed absurd. But for those in the field, it is a perfectly valid application of HCD and demonstrates just how narrow and superficial it is within industry and the academic world.

Genuine care for humanity would raise too many difficult questions for big business. It might put a spotlight on and challenge core business beliefs about the pursuit of economic growth, meeting sales targets by marketing products to people who could be better off without them, or the ethics of designing killing machines. Bezerra and Brasell-Jones (2005) go on to urge designers to look at the bigger picture, to follow a ‘humankind-centered design’ approach. But whereas they think companies will also benefit from this approach, I think it will, if pursued at all, put designers in sharper conflict with business. (And for this reason, it is unlikely that human-centred design will ever widen out to humankind-centred design in the business or academic world in any meaningful way.)

Interaction Designer as Businessman

Far from questioning business motives, many professional interaction designers have adopted the business mindset and seem all too eager to jump in and ingratiate themselves with business leaders. Designers who do this believe (or perhaps convince themselves) that what they have to offer (their design know-how) can benefit both the business and humanity. They do not associate business success with self-interested agendas and underhand tactics that are often contradictory to good design (such as dark patterns mentioned above). This thinking often leads designers into becoming champions of business – which also, unsurprisingly, finds them favour with business leaders and enhances their career prospects, which in turn bolsters their sense of accomplishment and self-belief.

Those who have ‘succeeded’ in this way are often happy to impart their views on what designers can do to better serve business. Consider this example from Jon Kolko’s (2010) Thoughts on Interaction Design where he describes how interaction design could become the key to business success:

Design is positioned to become a strategic differentiator for businesses looking for innovative differentiation, and thus the field is a likely evolution for many Industrial Designers. This strategic level of design is one that Interaction Design is prepared to participate in, and even own – if this type of designer is able to speak the common language of business and strategy. A great number of analysts have predicted just this sort of respect and strategic placement of design within traditional businesses. Daniel Pink, an author and business strategist, has been quoted repeatedly in major news publications as saying that “the MFA is becoming the new MBA.” (Kolko, 2010)

To his credit, Kolko does accept that interaction designers’ goals are often at odds with business priorities:

At the beginning stages of a project, an idea may be driven solely by a business necessity: increasing profits, gaining brand equity, or disrupting a traditional channel leader. The interaction Designer, if invited to discuss the project at this stage, may ask questions like “Does the user need this product at all?” This view might be informed by an understanding of culture, or an intricate care and love of society. [...] This is clearly a philosophical question first; the “right” answer may be the “wrong” business suggestion, and Interaction Designers are rarely invited to discuss the project at this stage. (Kolko, 2010)

It is interesting, however, that Kolko describes the question ‘Does the user need this product at all?’ as ‘philosophical’. The tendency to elevate such questions to philosophical levels is often a sign that we know what the simple, honest answer is but we want to avoid having to give it or act on it (in this case, because it would be bad for our employer). But if a designer, out of concern for society, gives an answer which happens to be the wrong ‘business suggestion’ then what does that tell us about the business? Moreover, thinking back to the parallel of corporation as psychopath, why would we even give any weight to what the business thought of the answer to that question?

Pursuing this conflict, we’ll surely find that there are many situations where to serve humanity and our conscience we will have to act against our employers’ interests. These differences cannot be resolved by designers forging closer ties with business, as Kolko goes on to recommend. They are fundamental to the way businesses are legally required to work. To raise these concerns with management is not a solution either – it will simply be interpreted as a sign of weakness or ignorance.

Yet concern for business priorities is very common in interaction design literature. Students will rarely find anything critical of corporations in their studies. Professionals in the field, many of whom write the books students are encouraged to read, are already in the business fold with many big corporate clients. They are also well versed in the language of business and go to great lengths to explain to students and other budding interaction designers what it takes to be successful in the business world. What’s missing is any kind of critical examination looking at how corporations themselves may be the obstacle to truly positive, socially responsible design.

Becoming a Radical Professional

The physicist and writer Jeff Schmidt (2001) has looked at the role of professionals in some depth in his excellent book Disciplined Minds. Schmidt argues that the professional’s lack of engagement with these issues is no accident. Professional training ‘produces obedient thinkers—highly educated employees who do their assigned work without questioning its goals.’ (Schmidt, 2001) Most professionals therefore end up posing no serious challenge to the status quo.

For designers, the dominant attitude can be summed up with the following quotes taken from interaction design literature: ‘Your job as a designer is to find a way to accomplish business goals’ (Goodwin, 2009); ‘It is the designers’ obligation to develop solutions without ever losing sight of … business goals’ (Cooper et al., 2007); ‘Design works best when there is a relationship of mutual trust and respect among Design, Business, and Engineering’ (Cooper et al., 2007).

Schmidt’s suggestion is for people to start questioning these assumptions and to begin working toward their own goals. Although difficult, Schmidt maintains that it’s possible to do it without eschewing corporate work. What’s needed is for concerned professionals to become ‘radical professional’ and ‘to work in the system without being of the system.’ Schmidt notes that there is some contradiction in the term ‘radical professional’ because a radical is someone who ‘undermines authority’ whereas a professional generally upholds it by ‘following assigned ideologies’. So to be a radical professional entails:

• Professionals seeing themselves as radicals first and professionals second.
• Holding a very critical view of the social role of the profession and of employers and clients.
• The radical professional’s politics making a difference in the world: ‘The fact that you are a radical makes no difference, and is therefore wasted, when you do things that would be done anyway.’ (Schmidt, 2001)

Schmidt goes on to suggest 33 practical things radical professionals can do on the job to advance their own goals for a better society. These acts may carry risks, and how many a radical professional can get away with depends on the individual and his/her place of work. Schmidt advises organising with co-workers and other like-minded people (not management) as much as possible. Below is an abbreviated list of 5 of Schmidt’s suggestions:

• ‘…help coworkers look beyond your employer’s public-relations image as serving the public. You help them to see the organization for the institution of the status quo that it is…’
• ‘…encourage coworkers to connect themselves with radical organizations and to read and subscribe to radical publications. You circulate antiestablishment periodicals, or selected articles from them, to professional and nonprofessional coworkers who might be interested.’
• ‘…assign your own curiosity. On the job, you develop and pursue your own goals while supposedly pursuing your employer’s goals. You steal as much time and as many resources as possible to do this. You encourage the hiring of more employees to give everyone more time to pursue their own goals.’
• ‘…give priority, during working hours, to helping coworkers with their own self-assigned, politically progressive projects.’
• ‘…blow the whistle and sabotage projects that are against the public’s interest. You act for a fairer social structure even at your employer’s expense.’

It should be clear from reading through this list that designers can make a difference while working inside a corporation, and not in the way usually suggested by many of today’s professional designers – many of whom are already in the corporate fold and who have internalised its values.

The field of interaction design is fast moving and the effects of many of the developments on ourselves and on society are unclear and often damaging. But those of us with real concerns about these developments should not be afraid to resist.

Acknowledgements

I would like to thank everyone involved with the Interaction Design Student Organisation (IDSO) at Chalmers University for discussing some of these issues with me and for making the study experience so much more enjoyable.

References

Bakan, J. (2005) The corporation: The pathological pursuit of profit and power. Constable. London

Bezerra, C and Brasell-Jones, M. (2005) Design responsibility in global open societies. In: University of Art and Design Helsinki, Joining Forces. Helsinki, Finland. 22-24 September 2005

Cooper, A, Reimann, R and Cronin, D. (2007) About Face 3. Wiley

Dark Patterns (2011) [online] [Accessed 16 February 2011]

Dunne, A, Raby, F. (2001) Design Noir: The Secret Life of Electronic Objects. Birkhäuser Basel

Eric Schmidt at Mobile World Congress. (2010) [video] [Accessed 20 February 2011]

Frank, R.H., Gilovich, T. and Regan, D.T. (1993) Does studying economics inhibit cooperation? Journal of Economic Perspectives. 7(2)

Fromm, E. (1997) To have or to be? Continuum

Fromm, E. (2003) Man for Himself. Routledge

Goodwin, K. (2009) Designing for the Digital Age Wiley

Hind, D. (2010) The return of the public. Verso

Kolko, J. (2010) Thoughts on interaction design. Morgan Kaufmann

Langley, C and Parkinson, S. (2009) Science and the Corporate Agenda: The detrimental effects of commercial influence on science and technology. Scientists for Global Responsibility

McKeon, M. (2010) The Evolution of Privacy on Facebook [online] [Accessed 16 February 2011]

Noble, D. (1995) Progress without people. Between the Lines

Papanek, V. (1985) Design for the real world. Thames & Hudson, London.

Schmidt, J. (2001) Disciplined minds. Rowman & Littlefield Publishers, Inc.

Schneier, B. (2010) Privacy and Control. [online] [Accessed 16 February 2011] .com/blog/archives/2010/04/privacy_and_con.html>

The Corporation. (2003) [Documentary] Directed by Mark Achbar and Jennifer Abbott. Canada. Big Picture Media Corporation

Automatic Extraction

Up to now we’ve relied mainly on PHP Readability to automatically identify and extract articles from web pages, and this is still how the majority of articles are extracted. It works extremely well for most pages, but there are still occasions when it fails – e.g. picks out the wrong HTML element, or doesn’t find anything at all. Improving PHP Readability will be one area of focus for future releases.

In 2.7 we still use PHP Readability, but we now recognise and prioritise hNews microformatting – if detected, we extract the first element marked entry-title and all elements marked entry-content. This is a standard that will hopefully be used more widely on the web. (For those who’ve asked, Twitter updates are now extracted properly because of hNews support.)

Site Patterns

Recognising that auto-detection does sometimes fail, in version 2.5 we introduced custom extraction patterns: a way for users to override auto-detection and tell Full-Text RSS (using CSS selectors) which element it should extract as the content block.

The biggest change in 2.7 is the introduction of site patterns. Site patterns sit in between custom extraction and auto detection. They allow fine grained control over extraction on a per-site basis. A site, identified by its domain name, can now have its own config file detailing extraction rules. Each time a URL is processed, we check to see if a corresponding site config exists, and if it does, we refer to it for instructions. Users can specify XPath expressions to match title and body elements and define rules to strip superfluous elements.

Rather than create our own configuration format for site patterns, we chose to adopt the same format used by Instapaper. Here’s what the entry for wikipedia.org looks like:

body: //div[@id = 'content']
strip_id_or_class: editsection
strip_id_or_class: toc
prune: no

Instapaper users will find these patterns by visiting instapaper.com/bodytext/ (login required).

One big advantage for us in using the same config format is that we can make use of all the existing rules listed on Instapaper. Marco, Instapaper’s creator, has opened up the database to allow for public contributions. So, included in Full-Text RSS 2.7 is over 100 site configuration files which will be applied automatically (look inside the site_config/standard/ directory). Most of these are borrowed from Instapaper, but we’ll soon be adding our own which we’ll be sharing with everyone.

Users can also create their own site config files and drop them in the site_config/custom/ directory. Each site config is simply a text file named after the site. For example, if I wanted a special rule for extracting content from this site, I would create a keyvan.net.txt file with the appropriate rules inside.

Extraction Process Overview

To summarise, Full-Text RSS 2.7 attempts to extract in the following order:

1. Custom Extraction Pattern
2. Site Patterns
3. hNews
4. PHP Readability

If at any stage we find we’ve got a successful title and body match, we do not proceed further. If, however, there is no match, we move down the list until there is (the only exception here is with custom extraction patterns – if the supplied CSS selector does not match, no further attempt is made).

Sound useful?

Full-Text RSS 2.7 is licensed under the AGPL and available to try or buy at fivefilters.org/content-only/.

Source: workersoftheworldrelax.org (via Medialens)

I was happy to learn last month that the plugin is being used on Golan Levin’s course Interactive Art & Computational Design at Carnegie Mellon University. See Processing Embedding.

The War You Don’t See – an excellent documentary by John Pilger — buy a DVD copy here.

Watch the rest: part 2, 3, 4, 5, 6