Posts Tagged ‘AutoGnome technology’

The Pragmatic Web: Preliminary Thoughts

Munindar P. Singh

Department of Computer Science

North Carolina State University

Raleigh, NC 27695-7535, USA



The core mission of the semantic web, namely, to enable information to be shared across

the web, faces significant challenges. The challenges come from the difficulty of capturing

semantics in a manner that is reusable across applications, the priority of process over data, the

importance of interaction, and the critical need for accommodating user context. Overcoming

these challenges takes us from the realm of semantics and brings us into the realm of pragmatics,

which we view as supervenient upon the semantics. We claim that when the vision of the

semantic web is realized, it will be via the pragmatic web.

The ideas described herein have been in development for a few years. Most recently, this work was supported by

the National Science Foundation under grant ITR-0081742 and by awards from IBM and Cisco. I have benefited from

discussions with several colleagues and students, especially including Mike Huhns, Mladen Vouk, Bin Yu, Zhengang

Cheng, Ashok Mallya, Mike Maximilien, Raghu Sreenath, and Pınar Yolum. Only I am responsible for the opinions

expressed, of course.

1 Introduction

In his contribution to the encyclopedia of unified science, Charles Morris popularized the science of

semiotics, namely, the study of symbolic systems [1938]. Morris defined semiotics as consisting of

three components: syntax, semantics, and pragmatics. Syntax deals with the structure of symbols,

semantics with their meanings, and pragmatics with their contexts of usage. These terms were

picked up by the early logicians and computer scientists and, especially the first two, are frequently

the objects of attention in computer science.

It is commonplace to view the web as a symbolic system. As its structural aspects have become

better understood, the semantics is drawing increasing attention. This position paper focuses on the

pragmatics that lies beyond the semantics of the web. Notice that just as the syntactic aspects have

not died off when the semantics is considered but have instead become more rigorously defined,

so too will the semantics flourish as the pragmatics expands. One might think of the pragmatics

endeavor as a special direction within the scope of the semantics endeavor now under way.

Interestingly, in common parlance, the word pragmatic also refers to practical, and in this case

the double meaning of the title is intended.

A brief history of data. In the early days of computing, data and applications were inextricably

intertwined leading, as we all know, to poor maintainability and upgradeability. By separating out

the data from the applications, database management systems enabled each to exist independently

of the other. The hope really was that data could be used for applications other than those for

which it was intended and that applications could access databases other than those they were

designed to access. This hope was partially realized in that, with sufficient effort, you can build

new applications that access old data and feed new data into old applications. However, in its

fullest form, this hope was dashed, because although access methods became almost standardized

on SQL for relational databases, the semantics of the data remained as ad hoc as ever.

The same problem occurred for the Internet. In the early days, applications and data formats

were ad hoc. The standardization on HTML enabled browsers through which people could access

information anywhere on the web. This is fine as long as humans are engaged in understanding

and processing the information, but doesn’t lend itself well to automation. So it is natural that

semantics will draw increasing attention on the web as well.

Pragmatic concerns. However, if there is one lesson to be learned from the long history of

databases, it is that it is practically impossible to describe data well enough for it to be used in

arbitrary applications. It will be worthwhile to take this lesson to heart. Doing so has some major

ramifications on how we should proceed in our investigations on the web.

 The current hype surrounding the semantic web is reminiscent of previous hype cycles associated

with artificial intelligence (AI). Like for AI, researchers seem to be dreaming up

scenarios where the semantic web could change how information on the web is prepared

and used. Each such scenario is within the scope of current technology to implement as a


demonstration system. However, when the scenarios are put together and expected to be automated

based on general-purpose representations, they pose a huge engineering challenge,

especially because current abstractions and theories are quite limited.

 The best hope for the semantic web is to encourage the emergence of communities of interest

and practice that develop their own consensus knowledge on the basis of which they will

standardize their representations. For example, such standards have emerged in narrow areas

of personal information management, e.g., with the vCard standard. Such standards have

also emerged in the handling of email attachments.

 The above will still remain inadequate, however, because it addresses the knowledge acquisition

and standardization challenges for the current focus on semantics, but ignores the

need for fundamental new representational frameworks. We cannot engineer the semantic

web merely by doing more of what we are doing with current buzzwords: XML, RDF, or

even DAML, because these after all no more than new syntax for ideas that originated in the

early days of work on data modeling and knowledge representation.

What we need is a special form of semantics, namely, pragmatics, which will have the additional

abstractions to enable the engineering of open information systems over the web.

Organization. This position paper develops its case by motivating services as essential to the

vision of the web, identifying technical challenges, and taking an initial cut at some principles to

address those challenges. Some of the arguments are admittedly exaggerated, but only for the sake

of making a point. Section 2 introduces our take on service composition, its necessity, and the

inadequacies of current approaches. Section 3 synthesizes the technical challenges from the study

of Section 2, identifies some principles for addressing them, and argues that multiagent systems

provide the right basis for an approach in which to realize those principles.

Although the main argument of this paper is new, some of the constituent claims have been

studied in our previous papers and columns on related subjects. Specifically, the following publications

are relevant and widely accessible: [Jain et al., 1999; Singh, 1998, 2000, 2001; Singh et al.,


2 Web Service Composition

The essence of the semantic web is to enable access to web services. Web services enable application

development and integration over the web by supporting program-to-program interactions.

The current examples of the semantic web might not involve a formal description of services, but

that is the obvious direction it is taking, and rightly so. In other words, when the semantic web is

realized and applied, it will mostly be through services.

Importance of composition. Although there can be some value in accessing a single service

through a semantically well-founded interface, the greater value is clearly derived through enabling


a flexible composition of services. Composition leads to the creation of new services from old ones

and can potentially add much value beyond just a nicer interface to a single service.

From a business perspective too, intermediaries that primarily offer access to a single service

would have a tough time thriving or even surviving. Airline travel agents are a case in point. Traditional

travel agents provide a nice user interface: friendly and with a human touch, but little

more. However, airlines do not like to pay commissions for services that merely repackage their

offerings. As a result, they compete with the travel agents and reduce their commissions, slowly

squeezing them out of the business. This is as one would expect where the offerings are conceptually

simple, especially for frequent customers. By contrast, package tour operators, who combine

offers from airlines and other vendors, can do all right. In other words, the increased complexity

due to subtle compositions is essential for intermediaries to flourish.

Describing and invoking services. It is natural to think of description and invocation as two

sides of the same coin. A description is how a provider offers its service; an invocation is how a

customer exercises an offered service. Web Services Description Language (WSDL) is a proposed

standard for describing services, essentially in terms of the method signatures they support [Christensen

et al., 2001]. Simple Object Access Protocol (SOAP) is a proposed standard for invoking

services, essentially through remote method invocations over HTTP [Box et al., 2000].

The DARPA Agent Markup Language (DAML) [Hendler and McGuinness, 2001] enables the

creation of ontologies in the description of specific web sites. It is not specifically geared to services,

but has obvious applications. Some emerging approaches add structure to service descriptions

through the use of ontologies, e.g., [Trastour et al., 2001]. Klein & Bernstein develop a richer

approach for describing and indexing services based on process models [2001].

Discovering services. Universal Description, Discovery and Integration (UDDI) [UDDI] is an

upcoming standard intended for announcing and discovering services. Despite its name, it doesn’t

offer support for integration. The idea is that services can be listed by their providers at a logically

central registry, which can be searched by prospective customer of service to discover a desirable

service. Because the indexing of the registry depends on how the services are described, this aspect

of discovery is intimately related to service description.

Describing compositions. No good means exist for describing desired compositions of services.

Currently, the best descriptions are procedural, typically based on previous work on workflow

management. For example,Web Services Flow Language (WSFL) was proposed to describe compositions

of services in the form of a workflow [Leymann, 2001]. WSFL specifications tell us

how different services ought to be invoked, e.g., in terms of ordering and parallelism among them.

DAML-S [Ankolekar et al., 2001] is a web service ontology from the DAML community. DAMLS

provides a core set of markup language constructs for describing the properties and capabilities

of web services. It also includes a traditional workflow-style process model for compositions of



3 Pragmatics

Below we identify some major challenges for semantic web service that are addressed neither by

current industry approaches nor by upcoming semantic approaches. Resolving these challenges

requires developing an approach that takes some principles seriously. The desired principles are

oriented toward the pragmatics of the web.

3.1 Challenges

Now we can consider the challenges for web services when they are geared toward composition.

Service description. An essential assumption of current approaches is that services can be described

in a manner that is independent of how they are used. However, services may not be fully

described through their methods. Descriptions given by service providers, even if using standardized

languages, might be incomplete. This is because of two reasons. One, specialized communities

of practice would use services in novel ways. Two, instead of a mere method invocation, the

appropriate metaphor is negotiation where the service provider and consumer may reason about

whether or not and, if so, how to interact with each other.

This becomes interesting and difficult when semantic exceptions are considered. Semantic exceptions

are not just about programming language or operating systems exceptions such as divideby-

zero or file-not-found, but deal with the meaning of the task at hand [Luo et al., 1998]. The

challenge is to handle exceptions in service compositions, not simply by throwing a programming

language exception, which is not necessarily the correct response. In any case, exceptions suggest

a case where the descriptions of services might evolve and might form an input to the questions of

compliance and trust mentioned below.

Service discovery and location. Current approaches implicitly assume that a logically central

registry of services can make the right matches for all comers. However, in real-life, prospective

service consumers would need to discover providers that are trustworthy in a manner that is sensitive

to their particular needs. A registry as an implicitly trusted third party seems unrealistic for

the following reasons. One, the registry has not interactions of its own to be able to judge the

quality or trustworthiness of a service provider or a prospective consumer. Two, what would matter

is the mutual trust placed in each other by a provider and consumer contemplating interacting

further. Three, the trustworthiness perceived by each in the other would depend upon their specific

interactions and contexts, something that a registry would not and should not be expected to model

for all participants. A proper solution would b social mechanisms for evaluating and discovering

trustworthy parties, both providers and consumers.

Interaction. While current approaches for service invocation represent much progress, they

carry the baggage of traditional distributed objects approaches. Current approaches are geared for

low-level invocation of services—they are not specially geared for enabling composition. Services

are integrated through method invocation without regard to any higher-level constraints. WSDL


allows us to capture the various methods but does not support constraints among those methods.

Either too many methods will always be enabled or too few. However, WSDL’s functionality is

required to specify the methods supported by a service.

Method invocation is appropriate for closed systems, but services are inherently autonomous

and often to be used in long-lived interactions. For example, a long-lived interaction occurs in

e-commerce when you try to change an order because of some unexpected conditions or try to

get a refund for a faulty product. Even short-lived settings involve protocols, e.g., checking if the

service requester is authenticated and properly authorized before accepting its order. Consequently,

interesting service compositions will often involve subtle constraints on interactions among the

participating services. The services won’t simply be invoked as methods by an aggregator but will

engage in rich protocols for potentially long-lived interactions.

Engineering composition. Current approaches take a procedural view of service composition

formulating workflow graphs that can be stepped through. Because of this, the main engineering

challenges that they bring up deal with standardizing on the data, e.g., through syntax or the

semantics. However, web services have attributes that set them apart from traditional closed applications:

they are autonomous, heterogeneous, long-lived, and interact in subtle ways to cooperate

or compete. Engineering a composition of such services requires abstractions and tools that bring

these essential attributes to the front. When the requirements are expressive, they highlight the

potential violations, e.g., the failure modes of a composition.

Engineering composition thus requires capturing patterns of semantic and pragmatic constraints

on how services may participate in different compositions. It also requires tools to help reject unsuitable

compositions so that only acceptable systems are built.

Compliance and trust. Because traditional approaches take a low-level, procedural stance on

service composition, they cannot deal with high-level interactions among services. Consequently,

they do not provide a basis for judging whether a particular service that is bound in the workflow

will in fact deliver the right interactions. Compliance is a lot more subtle than matching a method

signature. It should be rigorous even if context sensitive. Likewise, trust should be based on

interactions viewed in a fine-grained manner.

3.2 Principles

While developing community consensus to standardize the semantics for various domains seems

like a reasonable approach, we would still like to understand the principles through which web

applications and data can be linked. These principles lie not so much in the semantics as in the

pragmatics of the web. The shift to pragmatics has the following ramifications, which we formulate

as principles for building pragmatically sound web systems.

 User before provider. Instead of merely listing the methods or capabilities a service provider

might offer, it would help to model the needs of the consumers of the service. These consumers

will be presented architecturally as composers of services with potentially quite idiosyncratic

needs and composition requirements that would be highly context dependent.


The description, discovery, and invocation of services must be tied to the context of the

intended compositions.

 Process before data. The context of the usage of data is important and difficult to capture.

But understanding the processes behind the data give a better clue to its meaning than a

perfectly reusable (and therefore impossible to build) semantics would.

 Interaction before representation. Services provide an analog to a functional interface to

data. It is “analog” in that, as we argued above, services should go beyond method interfaces.

However, they provide a high-level interface that hides the irrelevant details of the semantics

of the data, while exposing the components of the semantics that are relevant for a particular

purpose. Just as functional interfaces to data structures hide their implementational details,

so too will interaction specifications of services hide the “excess” semantics of the data that

might otherwise be revealed.

Accommodating the above principles means giving a central position not to the data but to the

processes and contexts in which it is used. We can achieve the vision of the semantic web by first

concentrating on services rather than data and second by modeling services as agents rather than as

distributed objects and viewing service providers and consumers as participating in rich multiagent


3.3 Multiagent Systems

Agents are long-lived, persistent computations that can perceive, reason, act, and communicate

[Huhns and Singh, 1998a]. Agents act with varying levels of autonomy, depending on environmental

constraints and their ongoing interactions, e.g., as reflected in their previous commitments.

Because of their autonomy and heterogeneity, services can be naturally associated with agents.

Agents make it possible to capture interactions among services and the creation of new services as

subtle compositions of others.

Agents are not a panacea. However, applying agents enables us to naturally capture deeper

constraints on what services are willing to offer, capturing richer requirements for service composition,

discovering trustworthy services, negotiating within teams of providers, a basis for judging

the compliance of service providers with their contracts with respect to specific compositions.

4 Conclusions

Please note that our purpose is not to compete with or replace the existing offerings, but to augment

and refine them. Ultimately, we will need all the plumbing support and semantic descriptions that

existing approaches offer, but also a lot more.

This paper has raised more questions than it has answered. We close by invoking the metaphor

of Polya’s inventors paradox: “the more ambitious plan may have more chances of success” [1957].

The inventors paradox is the phenomenon in mathematics where a more general claim is easier to

prove, for example, when a stronger inductive hypothesis is required to prove the inductive step of


a proof by mathematical induction. In much the same way, we believe, the semantic web will be

the easier to realize if we follow it pragmatically, and that would be the obvious approach for any

pragmatic computer scientist.


Anupriya Ankolekar, Mark Burstein, Jerry R. Hobbs, Ora Lassila, David L. Martin, Sheila A.

McIlraith, Srini Narayanan, Massimo Paolucci, Terry Payne, Katia Sycara, and Honglei Zeng.

DAML-S: Semantic markup for web services. In Proceedings of the International SemanticWebWorking Symposium (SWWS), July 2001.

Don Box, David Ehnebuske, Gopal Kakivaya, Andrew Layman, Noah Mendelsohn, Henrik

Frystyk Nielsen, Satish Thatte, and Dave Winer. Simple object access protocol (SOAP)

1.1, 2000. http://www.w3.org/TR/SOAP.

Erik Christensen, Francisco Curbera, Greg Meredith, and Sanjiva Weerawarana. Web services

description language (WSDL) 1.1, 2001. http://www.w3.org/TR/wsdl.

James Hendler and Deborah L. McGuinness. DARPA agent markup language. IEEE IntelligentSystems, 15(6):72–73, 2001.

Michael N. Huhns and Munindar P. Singh. Agents and multiagent systems: Themes, approaches,

and challenges. In Huhns and Singh [1998b], chapter 1, pages 1–23. 1998a.Michael N. Huhns and Munindar P. Singh, editors.

Readings in Agents. Morgan Kaufmann, San

Francisco, 1998b.

Anuj K. Jain, Manuel Aparicio IV, and Munindar P. Singh. Agents for process coherence in virtual

enterprises. Communications of the ACM, 42(3):62–69, March 1999.

Mark Klein and Abraham Bernstein. Searching for services on the semantic web using process

ontologies. In Proceedings of the International Semantic Web Working Symposium (SWWS),

July 2001.

Frank Leymann. Web services flow language. TR WSFL 1.0, IBM Software Group, May 2001.

Zongwei Luo, Amit Sheth, John Miller, and Krys Kochut. Defeasible workflow, its computation

and exception handling. In Proceedings of the CSCW Workshop: Towards Adaptive Workflow

Systems, Seattle, 1998. At ccs.mit.edu/klein/cscw98/.Charles Morris, editor.

Foundations of the Theory of Signs. University of Chicago Press, Chicago

and London, 1938.

George Polya. How to Solve It: A New Aspect of Mathematical Method. Princeton Science Library.

Princeton University Press, Princeton, NJ, 2nd edition, 1957.


Munindar P. Singh. Agent communication languages: Rethinking the principles. IEEE Computer,

31(12):40–47, December 1998.

Munindar P. Singh. The service web. IEEE Internet Computing, 4(4):4–5, July 2000. Instance of

the column Being Interactive.Munindar P. Singh. Physics of service composition.

IEEE Internet Computing, 5(3):6–7, June 2001. Instance of the column Being Interactive.

Munindar P. Singh, Bin Yu, and Mahadevan Venkatraman. Community-based service location.

Communications of the ACM, 44(4):49–54, April 2001.

David Trastour, Claudio Bartolini, and Javier Gonzalez-Castillo. A semantic web approach to

service description for matchmaking of services. In Proceedings of the International SemanticWeb Working Symposium (SWWS), July 2001.

UDDI. UDDI technical white paper, 2000. http://www.uddi.org/pubs/Iru-UDDI-Technical-White-



Read Full Post »

The Pragmatic Web: Agent-Based Multimodal Web

Interaction with no Browser in Sight

Alexander Repenning & James Sullivan

University of Colorado, Computer Science Department, Boulder, USA


Abstract: To a large degree information has become accessible – anytime, anywhere – but not necessarily

useful. Unless the right information is presented at the right time, in the right place, and the right way

information may simply become unwelcome noise. The Pragmatic Web is about giving control to information

consumers so they can customize how to use information. The original Web – The Syntactic Web – has given

most of the control to the information producers but, aside of trivial font and plug-in options, has provided little

control to information consumers. The Semantic Web better separates content from presentation but has not yet

received general acceptance and lacks end-user customization tools. The Pragmatic Web – presented here – is

about providing control to information consuming end-users by enabling them to express computationally how

to turn existing information into personally relevant information.

Keywords: auditory input and output, context aware computing, end-user programmable agents, end-user

development and adaptation, sensors and actuators, virtual reality and 3D interfaces.

1 Introduction

Information is no longer a scarce resource. However,

this achievement is largely useless if information is

not provided in a format tailored to the user. Most

people in the United States, for instance, now have

access to online information thanks to inexpensive

computers and information appliances (Fox et al.,

2000), and public access to computing resources in

libraries and other institutions. Web-based

information can be accessed – through wires or

wirelessly – from desktop computers, laptops,

PDAs, cell phones, and specialized information

appliances. However, ubiquitous information accessdoes not imply universal information access. The

representational formats chosen by information

producers often do not match the needs of

information consumers. For instance, a Web page

may contain highly relevant information to a user,

but this information cannot be accessed if the user

cannot see or cannot read. Reasons for

producer/consumer information representation

mismatch include:

Wrong Modality: Blind users cannot read

textual descriptions. Automatic text-to-speech

interfaces may be able to verbally convey the

textual contents of a Web page to users, but if

the Web page is formatted for visual access, the

sequential presentation of information as speech

may be unintelligible or inefficient.

Wrong Language: Crucial explanatory text

may be provided in the wrong language. Less

than 15% of U.S. Web sites contain Spanish

translations (Lamb, 2001).

Wrong Nomenclature: Information may be

expressed in an unfamiliar measurement system.

The translation of Celsius to Fahrenheit or

kilometers to miles, while scientifically trivial,

may present a serious impediment to many users.

Wrong Time: Information may be correct,

relevant and readable, but presented at the wrong

time. Stock information, for instance, is most

useful when presented in real time.

Wrong Format: Information can look great on

a large computer monitor, but be completely

unsuitable for small information devices such as

PDAs and cell phones.

A mismatch between information presentation and

the formats required by an information consumer can

be difficult to address with a traditional Web

browser. For economic reasons, a producer may

choose to use a single representation scheme that

addresses only the needs of an anticipated majority

of information consumers. Because creating and

maintaining multilingual Web sites can be costly,

Figure 1: Distribution of Control over Representation

shared by Information Producer and Information


information consumers who are not proficient

English readers have few options with this model.

In the case of a person with a cognitive disability

planning to use the public transportation system,

there is a good chance that essential information

exists on the Web but cannot be accessed

meaningfully with conventional information

technology. The goal of this proposal is to extend

control to information consumers and let them use

information in fundamentally new ways that might

not be anticipated by information producers.

The ultimate questions involve who controlsinformation representation and how the information

is processed. The Control over Representation

diagram (Figure 1) illustrates a continuum of control

that ranges between two extreme positions. From

left to right, it identifies conceptual as well as

technical frameworks in order of increasing

information consumer control: the Syntactic Web,

the Semantic Web and the Pragmatic Web.

The Syntactic Web. In this first generation of Web

technology, a markup language (HTML) is used to

define content at a high level of detail. This

syntactic level controls the appearance of

information. Information producers define content,

font selection, layout, and colors. Information

consumers have limited control over representations

in their browser, including adjusting the size of

fonts, and enabling/disabling animations and plugins.

The Semantic Web. According to Tim Berners-Lee,

the Semantic Web (W3C, 2001, Berners-Lee et al.,

2001, Berners-Lee, 1999) will “radically change the

nature of the Web.” (Berners-Lee et al., 2001) The

formal nature of representation languages such as the

eXtensible Markup Language (XML) and the

Resource Description Framework (RDF) make Webbased

information readable not only to humans, but

also to computers. For instance, semantic-enabled

search agents will be able to collect machine-readable

data from diverse sources, process it and infer new

facts. Other research projects, such as the Avanti

project, have studied how to separate web content

from display modality to better serve the sensory

and perceptive abilities needs of users (Stephanidis

and Savidis, 2001).

Unfortunately, the full benefits of the Semantic

Web may be years away and will be reached only

when a critical mass of semantic information is

available. Critics of the Semantic Web (Frauenfelder,

2001) point out the enormous undertaking of

creating the necessary standardized information

ontologies to make information universally


The Pragmatic Web. In contrast to the Syntactic

and Semantic Web the Pragmatic Web is not about

form or meaning of information but about howinformation is used.

The Pragmatic Web’s mission is to provide

information consumers with computational agents totransform

existing information into relevantinformation of practical consequences. This

transformation may be as simple as extracting a

number out of a table from a single Web page or

may be as complex as intelligently fusing the

information from many different Web pages into

new aggregated representations.

This agent-based transformation needs to be

extremely flexible to deal with a variety of contexts

and user requirements. An agent running on a

desktop computer with a large display may utilize

rich graphical representation versus an agent running

on a cell phone with a small display may have to

resort to synthesized text information to convey the

same information.

The Pragmatic Web research explores the practice

of using information and the design of tools

supporting this process.

Instead of the traditional “click the link” browserbased

interfaces, agents capable of multimodal

communication will provide access to Web-based

information. Agent communication methods include

facial animation, speech synthesis, and speech

recognition and understanding. End-users or

caretakers will instruct agents to transform

information in highly customized ways. Agents will

work together to combine information from multiple

web pages, access information autonomously or

triggered by voice commands, and represent

synthesized information through multimodal


End-User Customization (Nardi, 1993, Jones,

1995) will be an integral part of the Pragmatic Web.

Successful development of end-user customization

will make giant steps toward an Every Citizen

Interface (ECI) (Committee et al., 1997) by letting

minority groups of information consumers, possibly

down to the single individual level, obtain ways to

control information representations in accordance

with their specific needs. This computer-supported

Information Processing is a form of knowledge

management (Sumner, 1999, Murray, 1999) that

turns raw data into information. End-user

customization will let users specify w h e r e

information is accessed (e.g. part of an existing Web

page), how it is accessed (e.g., voice activated), andhow information is further processed. For instance, a

Pragmatic Web application could run on a wireless

PDA equipped with GPS to help a person with a

cognitive disability navigate through town using the

local public transportation system.

The Pragmatic Web does not intend to subsume

the Syntactic Web or the Semantic Web. On the

contrary, the Pragmatic Web will initially work with

the Syntactic Web by letting end-user customizable

agents extract information out of existing (HTML)

Web pages (Inmon, 1996). When the Semantic Web

reaches a minimal critical mass, the Pragmatic Web

will then utilize the Semantic Web with agents that

access ontologies and make inferences based on these


This article describes how end-user programmable

agents allow users to change modalities to make

information show up at the right time, and to fuse

information from multiple sources into new formats.

To illustrate the Pragmatic Web framework, two

example applications are presented. The Mountain

Bike Advisor keeps track of current weather

conditions and personal biking preferences. A voice

interface recognises requests, sends out agents to

access remote sensor information and applies userdefined

rules to interpret information pragmatically.

The Boulder Transportation system tracks GPSequipped

busses in real time, renders a 3D

visualization, and interprets bus location information

to make navigational recommendations for persons

with cognitive disabilities.

2 Example Applications

Two early Pragmatic Web applications are presented

here to illustrate how agents are being programmed

by end-users, how these agents access information in

existing Web pages, how they process that

information, and finally how the interact with the

users by interpreting the information found. Both

applications are working prototypes.

2.1 Boulder Mountain Bike Advisor

This AgentSheets-based application connects realtime

Web information with speech recognition. A

user asks, “Where should I go mountain biking?”

Several agents located on a map of Boulder County

react to this voice command (Figure 2). These agents

are representing locations that are possible candidates

for biking and also feature real time, Web accessible

weather information sensors.

Figure 2: Boulder mountain bike


Rules, previously defined by the users, capture

pragmatic interpretations. For instance, an agent may

reply (using speech output): “It’s really nice up here

at Betasso but you should bring a jacket because it’s

a little windy.”

Behind the Scenes

How does all of this work? The information used by

the Bike Advisor originates in a number of Weather

stations featuring a large array of sensors accessible

via Web pages. The network of weather stations in

Boulder County, Colorado is relatively dense so that

for most bike trail locations a weather station can be

found to sufficiently well estimate current weather

conditions. The C1 Niwot Ridge Weather station is

closest to the Sourdough Bike Trail and features a

well-organized Web page (Figure 3).

Figure 3: NCAR’s Foothill Weather Sensor Web Page

serves as input for agents to make recommendations.

Our scenario begins with the user speaking the word

“Biking.” All the agents, represented as icons on the

map (Figure 3), listen to voice commands. Agents

in AgentSheets are programmed by end-users in

Visual AgenTalk (Repenning and Ambach, 1996).

Visual AgenTalk is a rule-based language. The first

rule of the Sourdough biking advisor agent has a

speech recognition condition becoming true as the

result of what the user just said. The agent now

triggers a second rule group called “check” which

includes two conditions accessing the C1 weather

station Web page to extract the current temperature

and wind speed information. The Fahrenheit

information is numerically converted into Celsius

information and, using text to speech, announced to

the user: “Temperature at Sourdough is currently

–3.2 degrees Celsius.”

Figure 4: AgentSheets end-user programmable

authoring environment illustrating if-then rules.

Temperature and wind speed are further interpreted

by calling a third rule group. This is the pragmatic

part of the interpretation using temperature and wind

thresholds that are only relevant to the user who has

expressed these rules. Unlike the objective part of

the rule which merely communicated the numerical

value of the temperature, the pragmatic part is

directly employed to reach a decision. In our case

since the temperature (in Fahrenheit) is less than 40

degrees the agent advises against a bike ride at

Sourdough: “It’s bitter cold up there. Don’t come

here!” In more moderate cases the agent would have

recommended to bring additional clothing such as

wind stoppers in case the wind speed exceeds a

different threshold.

There are two important things to note here. First,

the information returned by the agent is of a highly

pragmatic nature that is directly relevant to the user

and to the goal of mountain biking. These rules are

easily created and modified by the user using a

highly visual programming environment (Figure 4).

Second, the interface, input and output, can be

tailored to suit user needs and delivered in multimodal

forms, including visual and speech. For

example, the map display while helpful to locate a

specific trail, is not necessary to interact with the

system. Indeed the shift in modality from text to

speech allows the entire interaction to take place over

a cell phone without the need for any display or a

traditional browser, nor any need to modify the

original Web page in any way.

2.2 Mobility Agents

Navigating through a city public transportation

system can be a daunting challenge for a person with

memory and attention problems due to cognitive

disabilities. The Mobility-for-All research project

sponsored by the Coleman Institute for Cognitive

Disabilities is studying how personalized mobile

information technologies can assist such travelers by

eliminating information overload from traditional

navigational artifacts (Sullivan et al., 2002).

artifact Purpose

maps spatial relationships between one’s

current location and destination; identify

routing options; provide an abstract

means to assess overall trip progress.

schedules temporal information about route

availability at a given day and time.

landmarks to confirm global progress and anticipate

important events or tasks that will come

next, such as prepare to get off, etc.

labels and


to understand the local environment,

including: current location, where to meet

transportation vehicles; identify the

“right” vehicle; where to get on and off;

where to pay; etc.

clocks to synchronize schedules with physical

events, including transportation vehicle

arrivals and departures.

Table 1: Essential navigation artifacts commonly found

in public transportation systems

As buses travel on Boulder Colorado city streets,

they report their Global Positioning System (GPS)

locations on a wireless network. Agents track bus

locations, and use this information to generate

personally relevant “just-in-time” attention and

memory prompts in a visual and/or auditory form

that can be tailored to the mobile user’s needs,

abilities, and preferences.


Our current prototype consists of a number of

connected components (Figure 5). The existing

infrastructure includes the GPS-equipped

transportation system of 27 busses, transceivers that

connect the GPS sensors (one per bus) with a central

GPS information receiver, and a Web server that

puts the positional information onto the Web. Every

two seconds each bus sends an update of GPS

position, heading, identity and speed. Currently,

this information is gathered on an operations console

where it is both archived and visualized as dots

moving on a map. These dots are observed by a

human dispatcher, who informs drivers of bunching

and other problems. To a caretaker or our traveler, a

person with a cognitive disability, this

representation of information (at the syntactic level)

is without meaning or use.

Mobility Agents

The Mobility Agent server is the central architectural

component that mediates all communications.

Mobility Agents running on this server can read the

GPS information (bus ID and location) from the

provider server and track buses currently in the

transportation system. The server in turn feeds real

time bus and traveler information to the 3D

Transportation Situation Viewer.

When a traveler makes a choice, such as picking a

destination, the server receives that choice along

with the traveler’s location. Mobility agents generate

appropriate responses, which the server then

transmits to the traveler via events that contain

multimodal instructions, reminders, or prompts. The

server can also deal with traveler confirmations and

panic alarms.

The responses and information provided to the

traveler vary with the user profile, and depend on

mobility agents with custom settings for that

specific traveler. User profiles for multiple travelers

are maintained on the server and contain personalized

information such as schedules, typical itineraries and

destinations, and contact information. The behavior

of mobility agents can be readily modified, so the

server also provides caregivers customization

capabilities. The current version of the Mobility

agents uses Visual AgenTalk rules similar to the

ones shown previously to map data at the syntactic

level (the GPS location strings) to pragmatic level

(e.g., “your bus is here”).

Mobility agents produce signals that: 1) define

events available to the traveler; 2) create responses in

different modalities; and 3) provide exception

handling and error resolution mechanisms. A

caregiver then personalizes the mobility agents for

particular travelers by selecting settings appropriate

to travelers’ needs and abilities.

Figure 5: Mobility Agent Architecture

Real-Time Transportation Viewer

Caretakers see a real-time visualization of the

transportation state via the Real-Time Transportation

Situation Viewer. The client tracks the buses in the

transportation system as they run their routes. The

buses are depicted as moving 3D objects on a map

rendered in OpenGL. Bus stops are clearly marked

with superimposed bus stop signs. The travelers’

locations are tracked and represented on the 3D map

in real-time. The Mobility Agent server provides bus

and traveler location information.

Figure 6: A mobile agent-based prototype that provides

just-in-time prompts for people with cognitive

disabilities as they use public transportation systems.

Transportation Visualization lets caregivers monitor

travelers, assess a traveler’s ability to effectively use

the transportation system, detect difficulties in daily

routines, and receive emergency notifications.

Viewers can assume different perspectives in the 3D

word including fixed camera positions (e.g., at the a

bus stop), and cameras tracking objects (e.g., bus

driver, or traffic helicopter perspective).

Mobility Agent Customization Client

The Mobility Agent Customization Client would let

caregivers customize user profiles and mobility

agents through a browser interface. User profiles can

contain personal information such as typical

itineraries; such information can be used to detect

deviations from a normal route and evoke errorhandling

procedures. The Customization client

would provide an interface for customizing the

behavior of mobility agents to map specific

situations to appropriate actions. The client would

allow the specification of normal as well as

exceptional responses to traveler actions and choices.

For example, in a normal situation when everything

goes according to plan, the caregiver can define the

sequence of actions and prompts to be given to

Melanie, the cognitive disabled teen, after she

chooses her destination (home, in the case illustrated

by the scenario). This might include: 1) finding the

location of the right bus; 2) informing Melanie how

close it is to the bus stop she is at; 3) warning her

when it is arriving; 4) guiding her to board the bus;

5) informing her when her destination is

approaching; and 6) reminding her to gather her

belongings before leaving the bus. In an exceptionhandling

situation, such as Melanie boarding the

wrong bus, the caregiver can customize increasingly

intrusive prompts. These might include: first

warning her that she is on the wrong bus with subtle

sound notifications; then advising her to talk to the

driver; and then, if the situation is not resolved,

having the client application on the cell-phone send

a panic alarm to the caregiver. The caregiver could

then contact the traveler directly by phone. Events

and reactions are tailored to travelers with different

cognitive abilities. For example, an individual with

the ability to follow more complex instructions

might be advised to get off the bus, cross the street,

and catch a bus going in the other direction.

Wireless device

The client application running on a wireless device

accepts a traveler’s input choices and provides

instructions, reminders and prompts to the traveler

using multimodal means of communication, such as

voice input and output, sound, images, and movies.

Our current prototype is based on a cell phone

emulator running on a laptop equipped with a

wireless network. The prototype is functional in the

sense that it is connected to the Mobility agents,

receives information based on the actual bus

locations and allows users to define their goals.

The prototype illustrates agent-based components

of a mobile architecture (Figure 7). The person

simulated in this prototype (Melanie) is assumed to

be a teen with developmental disabilities including

attention and memory deficits. Melanie can be

“directed” to a bus stop where a bus is approaching,

and the prompting sequence is “triggered” by

selecting a destination option on her phone.

As the simulation runs, Melanie’s mobile phone

generates visual and auditory prompts triggered by

real world events. Prompts are generated to “get

ready” for her approaching bus, “please board now”

when the bus stops at her location, “please pull the

stop cord and prepare to get off” as the bus

approaches the destination stop, “please get off here”

at the destination stop, and finally, “don’t forget

your backpack.” As Melanie performs these tasks,

she is also rewarded with reinforcing praise.

The 3D visualization component could also

provide bus system operators or waiting passengers

with an overview of bus locations and status.

Observers can watch real time traffic, play back

recorded data or assume different camera perspectives

(e.g., birds-eye, bus stop perspective, bus driver

perspective). Bus users can locate relevant busses

based on their current position and bus identification

information. End-user development tools allows care

givers or service-providers, to specify rules that turn

the general bus information space into personally

relevant, pragmatic information communicated

through cell phones.

3 Discussion

A big concern with parsing general Web pages is

that the location or the format of a Web page may

change. Indeed this is a valid problem, which,

depending on the severity of the change and the

robustness of the parsing approach used, may result

in information that no longer can be accessed, or,

worse, may result in reporting the wrong

information. One way to think about the problem is

to assume that changes to Web information which

would confuse agents are also likely to confuse and

even upset real users. For instance, these days Web

designer are more careful with the creation of URLs

since they know that people share them in emails

and collect bookmarks. A Web site design strategy

that frequently changes the location of essential Web

pages is extremely likely to upset users.

A different approach to make parsing of Web

pages more robust is to factor out Web page

depending parts, which can be maintained separately.

AgentSheets agents can be freely distributed through

the Behavior Exchange (Repenning and Ambach,

1997). This way a community of service providers

can take over the responsibility of centrally

maintaining Web page depending agents. End-user

developers of agent-based applications may choose

to download the latest version of agents just in time.

In AgentSheets, for example, this is simple since

agents can download other agents.

The Semantic Web with its formalized interface

could provide a more robust approach. The

Pragmatic Web framework can and should use the

Semantic Web but at least for now may need to fall

back in most cases onto the Syntactic Web and more

informal parsing simply because the majority of

Web information is not yet available in semantic


4 Conclusions

The goal of The Pragmatic Web is to provide more

control to end-users with respect to using

information. Relatively simple end-user

programming techniques allow end-users to

Figure 7: A mobile architecture for locating and delivering traveler information – and finding help if something goes


transform general information available on the Web

into personally relevant information accessible via

wireless devices. End-user control – where to access

information, how to process information, how to

invoke information access, when to present

information, how to process information and how to

present information.

We have built a number of applications that,

using relatively simple end-user programs, have

wrapped up existing Web information in radically

different interfaces. The resulting shift is not about a

quantitative information access improvement but

about a qualitative shift in affordances. The

applications built are encouraging but should just be

considered simple instances of the much larger

Pragmatic Web framework.

5 Acknowledgements

This research has been supported by the National

Science Foundation (EIA 0205625, DMI 0233028)

and by the Coleman Initiative.

6 References

Berners-Lee, T. (1999) Weaving the Web: The

Original Design and Ultimate Destiny of

the World Wide Web by its Inventor,

Harper, San Francisco, CA.

Berners-Lee, T., Hendler, J. and Lassila, O. (2001)

The Semantic Web, Scientific American.Commission on Physical Sciences. (1997) More

Than Screen Deep: Toward Every-Citizen

Interfaces to the Nation’s Information

Infrastructure, National Academy Press,

Washington, D.C.

Fox, A., Johanson, B., Hanrahan, P. and Winograd,

T. (2000) Integrating Information

Appliances into an Interactive Workspace ,

IEEE Computer Graphics and

Applications, 30, 54-65.

Frauenfelder, M. (2001) A Smarter Web, TechnologyReview


Inmon, W. H. (1996) The Data Warehouse and Data

Mining, Communications of the ACM, 39,


Jones, C. (1995) End-User Programming, IEEEComputer, 28, 68-70.

Lamb, E. (2001) Web content struggles to go

worldwide , In Red Herring, Vol. 91, pp.


Murray, A. J. (1999) Knowledge management and

consciousness, Advances in Mind–Body Medicine, 16, 233-237.

Nardi, B. (1993) A Small Matter of Programming,

MIT Press, Cambridge, MA.

Repenning, A. and Ambach, J. (1996) Tactile

Programming: A Unified Manipulation

Paradigm Supporting Program

Comprehension, Composition and Sharing,

Proceedings of the 1996 IEEE Symposium

of Visual Languages, Boulder, Colorado,

pp. 102-109

Repenning, A. and Ambach, J. (1997) The

Agentsheets Behavior Exchange:

Supporting Social Behavior Processing, In

CHI 97, Conference on Human Factors in

Computing Systems, Extended

AbstractsACM Press, Atlanta, Georgia, pp.


Stephanidis, C. and Savidis, A. (2001) Universal

Access in the Information Society:

Methods, Tools, and Interaction

Technologies , International Journal of

Universal Access in the Information

Society, 1, 40-55.

Sullivan, J., Fischer, G., Binder, T. and Gregory, J.

(2002) Human-Centered Public

Transportation Systems for Persons with

Cognitive Disabilities – Challenges and

Insights for Participatory Design, In 7thParticipatory Design Conference (Ed,

Wagner, I.) Malmö, Sweden, pp. 194-198.

Sumner, M. (1999) Proceedings of the 1999 ACM

SIGCPR conference on Computer personnel

research, In Proceedings of the 1999 ACM

SIGCPR conference on Computer personnel

research New Orleans, LA USA.

W3C (2001), http://www.w3.org/

Read Full Post »

« Newer Posts

%d bloggers like this: