Watson News Companion

newscompanion screenshotWe recently ran a hackathon at work: people within IBM were invited to try building a mobile app aimed at consumers using Watson services. It was a fun chance to try out some new ideas, as well as to build something using our APIs – dogfooding is always a good thing.

I worked on a hack with David which we submitted on Wednesday. This is what we came up with, and how we built it.

The idea

A mobile app that will help users to digest the news by explaining references in stories and providing greater context.

Background

It’s difficult to find the time nowadays to properly read and understand what’s going on in the world. We rarely have the time to sit and read through a newspaper. Instead, we might quickly read news stories online from our smartphones and tablets. But that often makes it difficult to understand the broader context that a story is in. There might be references in the story to people, places, organisations or events that are unfamiliar.

Watson could help. It could be an assistant as you read the news, explaining unfamiliar references and the broader context.

Features

Our Watson News Companion demo is a mobile news reader app that:

  • anticipates questions and suggests areas where it can help improve understanding
  • provides answers to questions without needing the users to lose their place in the story
  • allow the user to dig deeper with their own follow-up questions


A video walkthrough of the hack

Implementation

The hack was built as a mobile web app using the MEAN stack: using express as the framework on a Node.js platform, storing some information in a MongoDB and building the UI with AngularJS.

It uses RSS feeds from news websites to fetch content, which are shown in a simple newsreader app built using Ratchet.

The contents of the story is run through the Watson Relationship Extraction API to pick out the people, places, organisations, and other entities mentioned in the story.

The API output includes co-references, to identify the multiple mentions of the same entity. These are combined and reviewed, and together with the type of the entity are used to generate likely questions about the entity.

These questions are sent to the Watson Question and Answer API. For questions which are returned with answers with a high confidence, annotations are added inline to the news story. Pressing the annotation brings up a sliding panel at the bottom of the screen with the answer to the question. The links and footer annotations are built using bigfoot.

Every screen in the app also includes an “Ask Watson” button which lets the user enters any free text question to let them dig deeper into what they’ve read.

Could you build this?

This was a proof-of-concept built in a hurry, so we’re not calling this a finished app. But everything we used is freely available – both to people inside IBM and the public.

We developed on an instance of Bluemix (our Cloud Foundry-based development platform) available internally within IBM. You can sign up for free to a public instance of Bluemix at bluemix.net.

The technologies used to build the hack are all freely available : Node.js, MongoDB, AngularJS, Ratchet, bigfoot, jQuery.

A beta version of the Watson Relationship Extraction API is freely available for apps hosted on Bluemix.

A beta version of the Watson Question and Answer API is freely available for apps hosted on Bluemix. But this is a demo instance of Watson that has only read a small number of general healthcare documents. That’s not a useful corpus for our hack, so to record our demo we stood up our own instance – using an untrained instance of Watson which we gave a small subset of Wikipedia to read. We used the Question and Answer API on this instance instead of the Bluemix one. For people outside IBM to do this, they need to sign up to join the Watson Ecosystem. This is also free, but there are criteria for who is eligible at this point, and an application process to go through.


Kids should learn to code

Does a five-year-old need to learn how to code?

A couple of weeks ago I was interviewed by the BBC. In a fairly long phone call, I either rambled inanely or provided detailed and nuanced answers in context. That depends on your point of view.

Either way, obviously not a lot of it could make it into their story, as they really only needed a few quotes. So I thought I’d put more of what I said here.

The background for the story was the changes to the UK school curriculum which means that all kids are being taught to code. And the basic premise for the piece was that as we’re “entering an era when computers are actually beginning to teach themselves” that this is unnecessary and that coding itself is becoming an outdated skill.

This is a summary of what I tried to say…

Learning to “code”

It’s useful to start with some context. When we talk about teaching kids to “code” we don’t just mean teaching them how to write lines of code – it’s broader than that. Some criticisms of this initiative seem to be arguing against five-year olds needing to learn where to put semi-colons, which is missing the point.

From what I’ve seen, it’s an umbrella term that covers a range of activities such as:

Logical thinking and problem solving

Teaching kids how to understand a description of a problem, identify a solution, and describe that solution by breaking it down into a series of steps.

As kids get older this can be framed as how to write an algorithm. But it’s something that can be started even at Faith’s age (6) and without needing to touch a keyboard. That’s not new – how many developers have had to answer the interview question “describe how to make a cup of tea”?

You don’t need to learn programming language syntax to start getting your head around this, and I would argue it’s a vital skill to develop in life, even if you don’t become a coder.

Technological creativity

We need to do more than teach children how to use the tools that they have today. We need to encourage an ethos from an early age that we don’t have to be passive users of technology.

It’s about teaching kids how to think of and how to approach technology. They don’t have to think of it as a black box that must be used as-is, but as something that they can remix and tweak and modify and change and create. It’s about an attitude of looking at technology as something that they can make do what they want to do, as opposed to use the way someone tells them they should.

This is what I love about running my Code Club. Instead of kids playing a random Flash game they find online, they can make a game themselves, the way they want it to be. If they want it to be faster, slower, bigger, smaller, a different colour, move differently: they are in control. It’s not fixed, they can make it do and behave the way they want it to. And if they realise that they can do that with technology, it’s a real light-bulb moment.

We need kids to have this mindset so they will grow up able to imagine the next wave of innovations. Saying that we don’t need this because we can delegate it to the computers we have today really feels to me to be missing the point. Cognitive computing holds exciting promise and potential but it does not mean “we won’t need to be creative any more, the computers will do that for us, too”.

Coding becoming “outdated”

Leaving aside this bigger picture, is coding itself a useful skill to learn. Is coding going to become outdated?

I don’t think so.

Part of this argument seemed to be “what is the point of teaching kids <insert-name-of-programming-language-here> because by the time they grow up it will be obsolete?”

Programming languages stick around longer than people think – there are people still making a living writing C and maintaining COBOL. (We’re normally after good Prolog people, too!)

But more importantly, a lot of what you learn in one language is transferable. Every time I’ve started working in a new programming language, I’ve built on the basic concepts I already know from others. Maybe we’ll teach children a programming language that isn’t the most widely used language when they’re older. But that doesn’t mean learning the underlying ideas will have been a waste of time.

The argument also seemed to be that not just any particular language, but coding in general will become obsolete. I’m not convinced by this.

What we mean by coding may be different in twenty years to what we mean today. In fact it probably will be. Coding will evolve. It always has, and I’m sure it will continue to.

Even just looking at my personal coding history, you can see that evolution. Writing in assembler (where I was moving data in and out of registers) was different to writing in C. And writing in C (where it wasn’t just about what I wanted it to do functionally, but also doing my own memory management) was different to my coding today in Java.

A big difference is in the level of abstraction. They all involved describing to the computer something that I wanted it to do. But the level of abstraction I’m able to use to describe it has changed.

I’m sure this is a trend that will continue. New programming languages will get higher and higher level. Future programming languages will give us ways to describe what we want with higher levels of abstraction. And maybe that will look closer to natural language than what we have today (well-written Java is already closer to being readable by a lay-person than assembler). Maybe it will be something like a Controlled English language that feels more like describing what you want to another person.

But that won’t mean that coding has become obsolete, just that it will have evolved as it always has.

The need for people who can understand a problem, and describe to a computer how to solve it, will remain – whatever language they use and whether that language looks like “code” as we understand it today.


Talking about IBM Watson (again)

As I mentioned in May, I was lucky to be able to go to Thinking Digital this year and talk about what we’re doing with Watson.

I’ve just noticed that they’ve made a video of my talk available. I haven’t dared watch it (does anyone like watching videos of themselves?), but I figured I should share it anyway!


Thinking Digital 2014

This week I went up to Newcastle for Thinking Digital.

It was the seventh Thinking Digital, but my first.

I’d seen a bunch of references to it being the UK’s answer to TED, the tickets aren’t cheap, videos from previous years look slick and professional, it’s held in The Sage which is a hugely impressive venue, they manage to get a great line-up of speakers, and the logistics in the run-up to the event were more organised than any event I’ve been to before.

So… I was expecting a cool and geeky, if faceless, serious, formal, and intimidating event.

I’d read it completely wrong. It’s absolutely a professionally run event. And there was no shortage of cool geekiness. But, more than that, the organizer, Herb Kim, has created a real sense of community in it. There’s a feeling of almost familial warmth amongst attendees who come year after year after year.

And they do it without being too cliquey. Everyone I spoke to was very friendly and welcoming, which made the few days a lot easier for an introvert like me. A few days being surrounded by and trying to talk to and socialise with several hundred smart brilliant people is the kind of thing I normally find hugely draining and more than a little daunting. But the crowd at TDC make it easier than most.

They value their time there, too. More than one person told me they’d paid for their own ticket and expenses to attend. I’m used to corporate-run conferences where everyone is paid for by their employer, or barcamps where people moan about being asked for a five pound deposit, so this surprised me.

The talks made for a fascinating and thought-provoking couple of days. I can’t do them justice here (when videos of the talks are available I’ll embed/link them here instead) but I want to give an idea of what the programme was like.

Jeni TennisonOpen Data Institute
Talked about the potential impact of open data on society, giving examples of how open data could be used to inform and widen access to debate.

Maik MaurerSpritz
Demonstrated their speed-reading technology – streaming one word at a time in a fixed place, for fast reading on mobile and wearable devices.

Gerard GrechTech City
Talked about the role of Tech City as a feedback loop between Government and the tech community.

Meri WilliamsChromeRose
Talked about the lessons that people managers could learn from artificial intelligence in how to inspire, motivate, and enable geeks to achieve great things.

Aral Balkanindie Phone
Gave an impassioned and stirring talk entitled “Free is a Lie” about the conflict between advertising-led business models, and user’s privacy and other interests.

David Griffithsfoam
Talked about using his background in the video game industry to combine crowd-sourcing and gaming to perform impressive citizen science projects.

Chi OnwurahMP for Newcastle Central
Talked about the parallels between technology and politics as driving forces for change, and the aims of the current Digital Government Review.

Mariana MazzucatoUniversity of Sussex
Argued that the image of the private sector as entrepreneurial and public sector as meddling and restrictive is an unhelpful myth and made the case for a bolder, entrepreneurial state.

Erin McKeanWordnik
Talked about the limitations of search as a model for accessing data and the need for discovery engines to find what you don’t know you want.

Blaise Aguera y ArcasGoogle
Described the history of machine intelligence and his predictions about what the future of machine intelligence might look like.

Carl LedbetterMicrosoft
Outlined the history and evolution of digital entertainment, and described the process that went into the design of the XBox One.

Jennifer GardyBC Centre for Disease Control
Described our progress in increasing our understanding of the human genome, and where it’s complexity lies.

Peter Gregson – Cellist
Gave a representation of the genome work that Jennifer had described. Instead of a data visualisation, it was a sonification. Using a cello.

Sean CarassoFalling Whistles
Told an inspiring story of how he came to learn about the terrible things happening in Congo, and how he went about trying to bring peace.

Conrad BodmanThe Barbican
Argued for recognition of the impact of digital tech on the arts, and described his projects to exhibit and showcase video games, animation, and digital effects.

Mark DearnleyHMRC
Described the challenges and need for technology in what HMRC do, and their digital ambition for the future.

Xavier De KestellerFoster + Partners
Talked about an amazing project to build a base on the moon, using autonomous robots with 3D printing heads to print a building out of moon dust.

Susan MulcahyImperial College London
Gave an energetic performance to describe the role of the red blood cell, and the science behind understanding brain injury.

Carlos UlloaHelloEnjoy
Showed what was possible using WebGL, bringing native 3D gaming to the browser without the need for plugins.

Jonathan O’HalloranQuantuMD
Described his work to create a mobile genetic-testing device, and the potential that real-time epidemiology from a mobile device could bring.

Blaise Aguera y ArcasGoogle
Talked about changes needed in society when more jobs are replaced by technology, and his observations about changes in gender dynamics.

Steve MouldBBC
Gave an entertaining talk about how he discovered, and tried to understand the science behind, the bead chain fountain.

Tom ScottUs Vs Th3m
Ended the conference with a fantastic performance showing what the impact of technology might be like in 2030.

Dale LaneIBM
And I did a Watson talk. I really didn’t want it to seem like a sales pitch, so I tried to put it in a bigger context of being a step forwards in changing how we use computers. I talked about why I work on Watson, what motivates and inspires me about it, and why I think what we’re doing is difficult but hopefully valuable. And I walked through a short demo to explain the value I see in where we are even now. Annoying technical issues (Keynote + clicker + multiple screens = fail) aside, it went okay. It was a lot to try and fit into 20 minutes, so I talked fast. :-)

Overall…

It was a fantastic event, and one I’d wholeheartedly recommend.

If you can get to a future Thinking Digital, you absolutely should.

It’s one of the most thought-provoking and interesting couple of days I’ve had in a long time.

.

Full-diclosure: As a speaker, I didn’t have to pay for a ticket to attend this event. My travel and accommodation costs were paid for by IBM.


Text analytics in BlueMix using UIMA

In this post, I want to explain how to create a text analytics application in BlueMix using UIMA, and share sample code to show how to get started.

First, some background if you’re unfamiliar with the jargon.

What is UIMA?

UIMA (Unstructured Information Management Architecture) is an Apache framework for building analytics applications for unstructured information and the OASIS standard for content analytics.

I’ve written about it before, having used it on a few projects when I was in ETS, and on other side projects since such as building a conversational interface to web pages.

It’s perhaps better known for providing the architecture for the question answering system IBM Watson.

What is BlueMix?

BlueMix is IBM’s new Platform-as-a-Service (PaaS) offering, built on top of Cloud Foundry to provide a cloud development platform.

It’s in open beta at the moment, so you can sign up and have a play.

I’ve never used BlueMix before, or Cloud Foundry for that matter, so this was a chance for me to write my first app for it.

A UIMA “Hello World” for BlueMix

I’ve written a small sample to show how UIMA and BlueMix can work together. It provides a REST API that you can submit text to, and get back a JSON response with some attributes found in the text (long words, capitalised words, and strings that look like email addresses).

The “analytics” that the app is doing is trivial at best, but this is just a Hello World. For now my aim isn’t to produce a useful analytics solution, but to walk through the configuration needed to define a UIMA analytics pipeline, wrap it in a REST API using Wink, and deploy it as a BlueMix application.

When I get a chance, I’ll write a follow-up post on making something more useful.

You can try out the sample on BlueMix as it’s deployed to bluemix.net

The source is on GitHub at github.com/dalelane/bluemixuima.

In the rest of this post, I’ll walk through some of the implementation details.

Runtimes and services

Creating an application in BlueMix is already well documented so I won’t reiterate those steps, other than to say that as Apache UIMA is a Java SDK and framework, I use the Liberty for Java runtime.

I’m not using any of the services in this simple sample.

Manifest

The app is bundled up in a war file, which is what we deploy. This is specified in manifest.yml.

Building

The war file is built by an ant task which has to include the UIMA jar in the classpath, and copy my UIMA descriptor XML files into the war.

I’m developing in eclipse, so I set up an ant builder to run the build, and configured the project to do it automatically.

I’m deploying from eclipse, too, using the Cloud Foundry plugins for eclipse.

XML descriptors

The type system is defined in an XML descriptor file and specifies the different annotations that can be created by this pipeline, and the attributes that they have.

Running JCasGen in eclipse on that descriptor generates Java classes representing those types.

The pipeline is also defined in XML descriptors: one overall aggregate descriptor which imports three primitive descriptors for each of the three annotators in my sample pipeline : one to find email addresses, one to find capitalised words and one to find long words.

Note that the imports in the aggregate descriptor need to be relative so that they keep working once you deploy to BlueMix.

These XML descriptor files are all added to the war file by being included in the build.xml with a fileset include.

Annotators

Each of the primitive descriptor files specifies the fully qualified class name for the Java implementation of the annotator.

There are three annotators in this sample. (XML files with names starting “primitiveAeDescriptor”).

Each one is implemented by a Java class that extends JCasAnnotator_ImplBase.

Each uses a regular expression to find things to annotate in the text. This isn’t intended to be an indication that this is how things should be done, just that it makes for a simple and stateless demonstration without any additional dependencies.

The simplest is the regex used to find capitalised words in WordCaseAnnotator and the most complex is the ridiculously painful one used to find email addresses in EmailAnnotator.

Note that the regexes are prepared in the annotator initializer, and reused for each new CAS to process, to improve performance.

UIMA pipeline

The UIMA pipeline is defined in a single Java class.

It finds the XML descriptor for the pipeline by looking in the location where BlueMix will unpack the war.

It creates a CAS pool to make it easier to handle multiple concurrent requests, and avoid the overhead of creating a CAS for every request.

Once the pipeline is initialised, it is ready to handle incoming analysis requests.

Once the CAS has passed through the pipeline, the annotations are immediately copied out of the CAS into a POJO, so that the CAS can be returned to the pool.

REST API

The war file deployed to BlueMix contains a web.xml which specifies the servlet that implements the REST API.

I’m using Wink to implement the API. The servlet definition in the web.xml specifies where to find the list of API endpoints and the URL where the API should be.

The list of API endpoints is a list of classes that Wink uses. There is only one API endpoint, so only one class listed.

The API implementation is a very thin wrapper around the Pipeline class.

Everything is defined using annotations, and Wink handles turning the response into a JSON payload.

That’s it

I think that’s pretty much it.

I’ve added a simple front-end webpage, with a script to submit API requests for people who don’t want to do it with something like curl.

It’s live at uimahelloworld.mybluemix.net.

Like I said, it’s very simple. The Java itself isn’t particularly complex. My reason for sharing it was to provide a boilerplate config for defining a UIMA analytics pipeline, wrapping it in a REST API, and deploying it to BlueMix.

Once you’ve got that working, you can do text analytics in BlueMix as complex as whatever you can dream up for your annotators.

When I get time, I’ll write a follow-up post sharing what that could look like.


Why am I still at IBM?

Ten years ago.

6 August 2003.

I was a recent University graduate, arriving at IBM’s R&D site in Hursley for the first time. I remember arriving in Reception.


Reception – the view that greeted me when I arrived

Ten years.

It was a Wednesday.

I’m still at the same company. I’m still at the same site. I still do the same drive to work, more or less.

For a *decade*.

How did that happen?

It was never The Plan. The Plan (as cynical as it sounds in hindsight) was that I’d stay for two or three years. I figured that would be long enough to get experience, and then I’d leave to work at a small nimble start-up which was where all the “cool” work was.

The Plan never happened. A few years passed, and then another few… I kept saying that I’d leave “later” and before I knew it a ten year milestone has kind of snuck up on me.

I think I’m more surprised than anyone. I’ve never been at any place this long. I was at Uni for five years. The longest I was at any school was four years.

It’s a serious commitment, and one I never realised that I had made. I’ve not even been married for as long as I’ve been with IBM.

So why? Why am I still here?

I live here.

It’s been varied

I’ve spent ten years working for the same company, but I’ve had several jobs in this time.

I’ve been a software developer. I’ve been a test engineer. I’ve been a service engineer, fixing problems with customer systems. I’ve worked as a consultant, advising clients about technology through presentations and running workshops. I’ve done services work building prototypes and first-of-a-kind pilot systems for clients.

I’ve written code to run on tiny in-car embedded systems and apps that ran on mobile phones. I’ve worked as a System z Mainframe developer. I’ve written front-end UI code, and I’ve written heavy-duty server jobs that took hours to run (even when they weren’t supposed to).


IBM Hursley

It’s still challenging

I’ve worked on middleware technology, getting some of the biggest computer systems in the world to communicate with each other, reliably, securely and at scale. I’ve used analytics to get insight from massive amounts of data. I’ve worked on large-scale fingerprint and voiceprint systems. I’ve used natural language processing to build systems that attempt to interpret unstructured text. I’ve used machine learning to create systems that can be trained to perform work.

I’m still learning new stuff and still regularly have to figure out how to do stuff that I have no idea how to at the start.


Some views of the grounds around the office

I get to do more than just a “day job”

I do random stuff outside the day job. I’ve helped organise week long schools events to teach kids about science and technology. I’ve mentored teams of University students on summer-long residential innovation projects. I’ve prepared and delivered training courses to school kids, school teachers and charity leaders. I’ve written an academic paper and presented it at a peer-reviewed research conference. And lots more.

I’m a developer, but that doesn’t mean I’ve spent ten years churning out code 40 hours a week. There’s always something new and different.


Hursley House – where I normally work when I have customers visiting

I work on stuff that matters

Tim O’Reilly has been talking for years about the importance of working on stuff that matters.

“Work on something that matters to you more than money”

If you’ve not heard any of his talks around this, I’d recommend having a look. There are lots of examples of his slides, talks, blog posts and interviews around.

I can’t do his message justice here, but I just want to say that he describes a big part of how I feel very well. I want to work on stuff that I can be proud of. Not just technically proud of, although that’s important too. But the pride of doing something that will make a difference.

Working for a massive company gives me chances to do that. I’ve worked on projects for governments, and police forces, and Universities. I’ve done work that I can be proud of.

For the last couple of years, I’ve been working on Watson. It’s a very cool collection of technologies, and watching the demo of it competing on a US game show has a geeky thrill that doesn’t get old. But that’s not the most exciting bit. Watson could be a turning point. This could change how we do computing. If you look at what we’re trying to do with Watson in medicine, we’re trying to transform how we deliver healthcare. This stuff matters. It’s exciting to be a part of.


These are the views that surround the site

I like the lifestyle

Hursley is a campus-style site. It’s miles from the nearest town, and surrounded by fields and farms. It’s quiet and has loads of green open space.

My commute is a ten minute drive through a village and fields.

I don’t have to wear a suit, and I don’t stand out coming to work in a hoodie and combat trousers. Flexitime has been the norm for most of my ten years, and I am free to plan a work day that suits me. When I need to be out of the office by 3pm to get the kids from school, I can.

My kids are at a school half-way between home and the office, so I can do the school run on the way to work. As the school is only five minutes from work, I often nip out to see them do something in an assembly, or have lunch with them.

This is a nice aspect of the school – that parents are welcome to join their kids for lunch, and have a school dinner with them and their friends in the school canteen. But still… it’s pretty cool, and if I didn’t work just up the road from them, I wouldn’t be able to do it.

Once a month, I bring them to work in the morning before school starts for a cooked breakfast in the Clubhouse with the rest of my team.

All of this and a lot more tiny aspects like it add up to a lifestyle that I like.

More train tickets
Some of my train tickets from the last few years

I get to see the world

I enjoy travelling. I love seeing new places.

But I’d hate a job where I lived out of a suitcase and never saw the kids.

I’ve managed to find a nice balance. I travel, but usually on short trips and not too often.

In 2006, I worked at IBM’s La Gaude site near Nice. In 2007, Singapore, Malaysia, Philippines and Paris. In 2008, I worked in Copenhagen, Paris and Hamburg. In 2009, I worked in Munich many times, and Rotterdam. In 2010, Stockholm. In 2011, Tel Aviv and Haifa in Israel, Austin in Texas, Paris and Berlin. Last year, I worked in Zurich and Littleton, Massachusetts.

This year, I’ve been in Rio de Janeiro and Littleton again, and it looks like I’ll be in Lisbon in December.

Plus working around the UK. It’s less glamorous, but it’s still interesting to go to new places. I’ve worked in loads of places, like Edinburgh, York, Swansea, Malvern, Warwick, Portsmouth, Cheshire, Northampton, Guildford… I occasionally have to work in London, although I tend to moan about it. And I spent a few months working in Farnborough. I think I moaned about that, too. :-)

Travelling is a great opportunity. I couldn’t afford to have been to all the places that IBM has sent me if I had to pay for it myself.

The officemy deskCarnage
My office today (left), compared with some of the other desks I’ve had around Hursley

The pay is amazing!

Hahahahaha… no.

See above.


Grace at my desk at a family fun day at work in 2008

Will I be here for another ten years?

I’m trying to explain why I’m happy and enjoying what I do. I’m not saying I couldn’t get exactly the same or better somewhere else. Because I don’t know. Other than a year I spent as an intern at Motorola I’ve never worked anywhere else. For all I know, the grass might be greener somewhere.

Will I still be here in another ten years? I dunno… I do worry if that’s unambitious. I wonder if I should try somewhere else. I wonder if only ever working for one company is giving me an institutionalised and insular view of the world.

I keep getting emails from LinkedIn about all the people I know who have new jobs. There are a bunch of people I used to work with at IBM who have not only left to work at other companies, but have since left those companies and gone on to something even newer. While I’m still here.

Am I destined to be one of those IBMers who works at Hursley forever? That’s a scary thought.

For now, I’m enjoying what I do, so that’s good enough for me.

Happy 10th anniversary to me.


W4A : Accessibility of the web

This is the last of four posts sharing some of the things I saw while at the International World Wide Web Conference for w4a.

Several presentations looked at how accessible the web is.

Web Accessibility Snapshot

In 2006, an audit was performed by Nomensa for the United Nations. They reviewed 100 popular websites for conformance to accessibility guidelines.

The results weren’t positive: 97% of sites didn’t meet WCAG level 1.

Obviously, conformance to guidelines doesn’t mean a site is accessible, but it’s an important factor. It’s not sufficient, but it is required. Conformance to guidelines can’t prove that a website is accessible, however there are some guidelines that we can be certain would break accessibility if not followed. So they are at least a useful starting point.

However, 2006 is a long time ago now, and the Internet has changed a lot since. One project, from colleagues of mine at IBM, is creating a more up to date picture of the state of the web. They analysed a thousand of the most popular websites (according to Alexa) as well as a random sampling of a thousand other sites.

(Interestingly, they found no statistically significant difference between conformance in the most popular websites and the randomly selected ones).

Their intention is to perform this regularly, creating a Web Accessibility Snapshot, with regular updates on the status of accessibility of the web. It looks like it could become a valuable source of information.

Assessing accessibility

There was a lot of discussion about how to assess accessibility.

One paper argued there is an over-reliance on automated tools and a lack of awareness of the negative effects of this. They demonstrated a manual review of websites, comparing results with output from six popular tools. Their results showed how few accessibility problems automated tools discover.

Accurately assessing a website against accessibility guidelines doesn’t necessarily mean that you can prove a site is accessible or easy to use.

Some research presented suggests guidelines only cover a little over half of problems encountered by users. Usability studies suggest some websites that don’t meet guidelines may be easier to use than websites that do, as users may have effective coping strategies for (technically) non-compliant sites. This suggests we need a better way of assessing accessibility.

A better approach might be to observe users interact with a website and assess based on their experiences. One tool presented, WebTactics, showed an automated approach to assessing accessibility by observing a user and identifying behaviours they employ.

Another paper detailed how to add accessibility monitoring to a live website by adding additional JavaScript that captures and evaluates mouse clicks and button presses client-side before submitting them to a server for processing. Instead of requiring the user to perform predefined, and perhaps artificial, tasks, they hope to be able to discover tasks implicitly – that common tasks will emerge from the low-level actions that they collect.

Accessibility training

Given that most websites have some sort of accessibility problems, there was some talk about how this could be improved.

One project presented showed training that has been developed to raise awareness of how people with disabilities access the web, and the implications of the accessibility guidelines. It’s a practical course including hands-on assignments, and looks like it could be the sort of thing that could help web developers make a real difference.

Social Accessibility

Another project is using crowd-sourcing to improve web sites that already exist. Social Accessibility, another IBM project, enables volunteers to make web pages more accessible to the visually impaired.

It provides a mechanism for accessibility problems to be gathered directly from visually impaired users. Volunteers are then notified, and can respond using a tool that allows them to externally modify web pages to make them more accessible. It lets them publish metadata associated with the original web page. This can be applied to the web page for all visually impaired users who visit it in future using this tool, so that many users can benefit from the improvement.

cloud4all

Finally, a project called cloud4all is developing a roaming profile that stores your preferences in a way that multiple services can access. The focus is on accessibility – a user can store their accessibility needs in one place, and then interfaces can use this to adapt for them.


Dyslexia at W4A

This is the third of four posts sharing some of the things I saw while at the International World Wide Web Conference for w4a.

There were a few sessions presenting work done to improve understanding of how to better support people with dyslexia.

One interesting study investigated the effect of font size and line spacing on the readibility of wikipedia articles.

This was assessed in a variety of ways, some of which were based on the reader’s opinions, while others were based on measurements made of the reader during reading and of their understanding of the content after. The underlying question (can we make Wikipedia easier to read for dyslexics?) was compelling. It was also interesting to see this performed not on abstract passages of text, but in the context of using an actual website.

Accessibility isn’t just about the presentation but also the content itself. Another study looked at strategies for simplifying text that could make web pages more readable for dyslexic readers.

It compared the effectiveness of two strategies: firstly, providing synonyms on demand – giving a reader a way to be able to request an alternative for any word. The second was providing synonyms automatically – with complex words automatically substituted for simpler equivalents. Again, this was assessed in several ways, such as the speed of reading, the reader’s comprehension, on the reader’s opinion of easiness, on the effort it took (e.g. interpreting facial expression, etc.), on fixation duration measured using eye tracking, and so on.

On a more practical note, there were also tools presented that are being created to help support people with dyslexia.

Firefixia is a Firefox toolbar extension being created by colleagues of mine in IBM. It provides options for users to customise the web page they are looking at, offering modifications that have been demonstrated to make it easier for dyslexic users.

Dyseggxia is an impressive looking iPad game that aims to support children with dyslexia through fun word games.


W4A : Future of screen readers

This is the second of four posts sharing some of the things I saw while at the International World Wide Web Conference for w4a.

Several of the projects that I saw showed glimpses of a possible future for screen readers.

I’ve written about screen readers before, and some of the challenges with using them.

Interactive SIGHT

One project interpreted pictures of charts or graphs and created a textual summary of the information shown in them.

I’m still amazed at this. It takes a picture of a graph, not the original raw data, and generates sensible summaries of what it shows.

For example, given this image:

It can generate:

This graphic is about United States. The graphic shows that United States at 35 thousand dollars is the third highest with respect to the dollar value of gross domestic product per capita 2001 among the countries listed. Luxembourg at 44.2 thousand dollars is the highest

or

The dollar value of gross domestic product per capita 2001 is 25 thousand dollars for Britain, which has the lowest dollar value of product per capita 2001. United States has 1.4 times more product per capita 2001 than Britain. The difference between the dollar value of gross domestic product per capita 2001 for United States and that for Britain is 10 thousand dollars.

The original version was able to process bar graphs, and was presented to W4A in 2010. What I saw was an extension that added support for line graphs.

Their focus is on the sort of graphics found in newspapers and magazines – informational, rather than scientific graphs. They want to be able to generate a high level summary, rather than a list of plot points that require the user to build a mental model in order to interpret.

For example:

The image shows a line graph. The line graph presents the number of Walmmart’s sales of leather jackets. The line graph shows a trend that changes. The changing trend consists of a rising trend from 1997 to 1999 followed by a falling trend through 2006. The first segment is the rising trend. The rising trend is steep. The rising trend has a starting value of 1890. The rising trend has an ending value of 36840. The second segment is the falling trend. The falling trend has a starting value of 36840. The falling trend has an ending value of 12606.

The image shows a line graph. The line graph presents the number of people who started smoking under the age of 18 in the US. The line graph shows a trend that changes. The changing trend consists of a rising trend from 1962 to 1966 followed by a falling trend through 1980. The first segment is the rising trend. The rising trend is steep. The second segment is the falling trend.

It’s able to interpret an image and recognise trends, recognise how noisy or smooth it is, recognise if the trend changes, and more. Impressive.

Interpreting data in tables

Another project demonstrated restructuring data tables in web pages to make them easier to explore with a screenreader.

They have an interesting approach of analysing an HTML table and reorganising it to make it more accessible, abstracting out complex sections into a series of menus.

For example, given a table such as this:

it can produce navigable menus such as this:

Even quite complex tables, with row and column spans, which would otherwise be quite difficult to interpret if read row-by-row by a screenreader, is made much more accessible.

Capti web player

Another technology I saw demonstrated was the Capti web player.

Tools such as instapaper and read it later have showed that we can take most web pages and extract the body text for the article on the page.

This capability should be ideal for visually impaired users, but the tools themselves are still quite difficult to use and integrate poorly with assistive technologies. Someone described them as obviously “designed by sighted people for sighted people”.

Capti combines this capability with an accessible media player making it easy to navigate through an article, move through a list of articles, and so on. To a sighted user like me, it looked like they’ve mashed together instapaper with an audiobook-type media player. I often listen to podcasts while I go running, and am a heavy user of pocket and Safari’s reading list. So this looks ideal for me.

Multiple simultaneous audio streams

Finally, one fascinating project looked at how to make it quicker to scan large amounts of content with a screenreader to find a specific piece of information. I’ve written before that relying on a screenreader (which creates a sequential audio representation of the information on the page, starting at the beginning and going through the contents) can be tremendously time-consuming, and that it results in visually impaired users taking considerably more time to find information on the web.

This project investigated whether this could be improved by using multiple simultaneous sound sources.

It sounds mad, but they’re starting from observations such as the cocktail party effect – that in a noisy room with several conversations going on, we’re able to pick out a specific conversation that we want to listen for. Or that a student not paying attention in a lecture will hear if a lecturer says something like “this will be on the exam”.

They’re looking at a variety of approaches, such as separating the channels directionally, so one audio stream will sound like it’s coming from the left, while another is in front. Or having different voices, such as different genders, for the different streams. It’s an intriguing idea, and I’d love to see if it could be useful.


Web technologies I saw at W4A

WWW2013

Last month I went to the International World Wide Web Conference for w4a. I saw a lot of cool web technologies and accessibility projects while I was there, so thought I would share links to some of the more interesting bits.

There are too many to put in a single post, so I’ll write a few posts to cover them all.

Subtitles

Subtitles and transcripts came up a few times. One study presented looked at online video, comparing single-line subtitle captions overlaid on the video with multi-line off-screen transcripts adjacent to it.

It examined which is more effective from a variety of perspectives, including readability, reader enjoyment, the effect on understanding and so on. In summary, it found that overlaid captions are generally better, although transcripts are better for content which is more technical.

Real-time transcription from a stenographer at W4A

We had subtitles for all the talks and presentations. Impressively, a separate screen projected a live transcription of the speaker. For deaf attendees, it allowed them to follow what the speaker was saying. For talks given in Portuguese, the English subtitles allowed non-Portuguese speakers like me to understand.

They did this by having live stenographers listening to an audio feed from the talks. This is apparently expensive as stenography is a skilled expertise, and it needs to be scheduled in advance. It’s perhaps only practical for larger conferences.

Legion Scribe

This was the motivation for one of the more impressive projects that I saw presented : Legion Scribe, which crowd-sourced real-time captioning so that you wouldn’t need an expert stenographer.

Instead, a real-time audio stream is chopped up into short bits, and divided amongst a number of people using Mechanical Turk. Each worker has to type the short phrase fragment they are given. The fragments overlap, so captions that each worker types can be stitched back together to form captions for the whole original audio stream.

All of this is done quickly enough to make the captions appear more or less in real-time.

Seriously impressive.

And they’re getting reasonable levels of coverage and accuracy. The system has been designed so that workers don’t need to be experts in the domain that they’re transcribing, as they’re only asked to type in a few words at a time not whole passages. With enough people, it works. If they have at least seven workers, it’s approaching the coverage you can get with a professional stenographer.

Assuming that Mechanical Turk can provide a plentiful supply of workers, then this would not only be cheaper than a stenographer, but also let you start captioning at a moments notice, rather than needing to arrange for a stenographer in advance.

Map Reduce in the browser

Speaking of crowd-sourcing, the idea of splitting up a large computing task between a large number of volunteer computers isn’t new. SETI@home is perhaps the best known, while World Community Grid is a recent example from IBM.

But these need users to install custom client software to receive the task, perform it and submit the results.

One project showed how this could be done in web browsers. A large computing task is divided up into map reduce jobs, which are made available through a website. Each web browser that visits the website becomes a map reduce worker, running their task in the background using web workers. As long as the user remains on the site, their browser can continue to contribute to the overall task in the background, without the user having had to install custom client software.

It’s an elegant idea. Not all sites would be well suited to it, but there are plenty of web sites that I keep open all day (e.g. GMail, Remember The Milk, Google Calendar, etc.) so I think the idea has potential.

Migrating browser sessions

An interesting project I saw showed how the state of a browser app could be migrated from one browser to another, potentially a different browser running on a different machine even a different platform.

This is more than just the client-server session, which you could migrate by transferring cookies. They’re transferring the entire state of dynamic AJAX-y pages: what bits are open, enabled, and so on, for any arbitrary web app.

Essentially, they started by wanting to be able to serialize the contents of window, so that it could transferred to another browser where it could be used to restore from.

That wouldn’t be enough. window doesn’t have access to local variables in functions, it wouldn’t have access to most event listeners such as those added with addEventListener, it wouldn’t have access to the contents of some HTML5 tags like canvas, it wouldn’t have access to events scheduled with setTimeout or setInterval, and so on.

Serializing window gets you the current state of the DOM which is a good start, but not sufficient to transfer the state for most web apps.

A prototype system called Imagen shows how this could be done. Looking at how they’ve implemented it, they’ve had to resort to using a proxy server which intercepts JavaScript going to the browser and instruments it with enough additional calls to let them access all of the stuff that wouldn’t normally be in scope. This is enough for them to be able to serialize the entire state of the page.

I can see a lot of uses for this, such as in testing, debugging or service scenarios, as well as just the convenience of being able to resume work in progress as you move between devices.

Inferring constraints on REST API query parameters

Many web services include constraints and dependencies for the query parameters. For example: “this option is always required”, “that parameter is optional”, or “you have to specify at least one of this or that”. For example, the twitter API docs explain how you have to specify a user_id or screen_name when requesting a user timeline.

One project I saw was an attempt to automatically infer these rules and dependencies through a combination of natural language processing to recognise them in API documentation, and automated source code analysis of sample code provided for web services. It combines these into an estimated model of the constraints in the REST APIs, which are then verified by submitting requests to the API.

They demonstrated it on APIs like twitter, flickr, last.fm, and amazon, and it was surprisingly effective.

duolingo

Finally, there was a keynote talk on Wednesday by the founder of duolingo.

Captcha is particularly interesting because it uses a task that people need to do anyway (verify that they’re human) to crowd-source the completion of a task that needs to be done (digitise the text of old books that cannot be read by automated OCR).

Duolingo is similar. It takes a task that people need to do, which is to learn a new language, and uses that effort to translate texts into different languages.

It’s better explained by their demo video.

It’s been around for a little while, but I’d not come across it before. Since getting back from www, I’ve been trying it out. Even Grace has been using it to improve her French and seems to be getting on really well with it.

What else?

There were a lot of other cool projects and technologies that I saw, so I’ll follow this up with another post or two to share some more links.