Mass Collaboration Workshop, Day Three
Darren Gergle
Understanding and Bridging the Wikipedia Language Gap
There is an extensive literature on Wikipedia, everything from collaboration to participation to embedded bias to the uses of Wikipedia data structures by AI and natural language processing application.
So, in different places people speak different languages, but the presumption is people are reporting on the same things, and have the same sort of coverage biases. But as we consider the gloabl nature of information we need to think about equitable information access, retaining diverse perspectives, and algorithmic biases and the role they play in information structure, representation, and consumption. A part of this talk is to look at tools that address this.
Gerhard Fisher presented a nice picture of how things are - this presentation is more about how things should be.
To what extent is there content diversity among Wikipedia language editions? I thought this would be a simple question - five years ago. There are two aspects here in the implicit assumption that every language describes the same sorts of things, and so there is a global consensus among the concepts. But there is diversity at the concept and sub-concept levels.
We have concept diversity - that is, the set of titles of articles.
We have sub-concept diversity - that is, the set of topics within a given article.
First of all, the methodology, matching pages to concepts. So we want (eg.) to find a core concept ('chocolate') and then match the pages in different languages to this topic. There are 'language links' on the wikipedia pages that will take you to the other language for the same page. We take a hyperlingual approach, which gives us a cluster of pages (sometimes even if the link between two language editions is missing).
The 'conceptual drift' problem also occurs - the boundaries of conceptions changes across language. So, eg., we have a page in English on 'river', which links to a German page, which links bacl to 'canal', which links to a French page, which links back to 'canyon'. So the algorithm has parameters to tune the tightness of concepts - it first limits the number of edges from a given language, and second, has a minimum number of languages to allow an edge to remain.
So after examining the concepts we discovered that the archetypical concept looks nothing like 'chocolate'. They look more like a region or place or person (explore this?) We also find that the bulk of concepts are single-language concepts, with only a small number of concepts common to even three languages.
So - why? Maybe it's just that English, being the largest, is a superset. And everything else is just growing into it. But when we do pairwise comparisons this doesn't bear out - comparing English to German, for example, we find 40 percent of the concepts unique to each language.
So - analyzing the concepts - we can (Adafre and de Rijke, 2006) use the links on the page to create a conceptual representation of the page. Now we can compare these between languages. So we look at which links they have in common, which defines an 'overlap coefficient'. The mean OC is 0.41 - that is, about 59 percent of the links in the shorter articles (in one language) don't appear in the larger articles (in another language).
So, what accounts for this diversity?
We look at the concept of self-focus - the degree to which language-based cultures centre their descriptions around objects and entities of regional importance. So we used geo-tags to examine this, drawing out articles with a spacial location. We use a technique called 'spatial indegree sums' - so, for example, we take an area like 'Poland', find the entities within that region, and then find all the articles that link to those original entities. So we can give a particular weight to a given geographic area. This allows us to compare an indegree sum for different languages for a given geographic area, which allows us to measure self-focus.
So - the global consensus hypothesis would suppose that each region has relatively equal indegree sums for each region, while the diversity hypothesis would predict that the polish-language Wikipedia would favour Poland. The actual results show regional diversity - people regionalize the information around them. The English Wikipedia focuses on the US and Britain, the Russian Wikipedia on Russia, etc.
So: self-focus is a systemic boas in Wikipedia. people orient the knowledge and language around themselves. This diversity is a serious concern for semantically based AI and NLP applications.
On the converse side: we could create applications that take advantage of this diversity. We built a system called Omnipedia that provides access to 85M+ concepts in 25 languages. Pick a concept - it displays the number of related links for each of the languages, as nodes: you click on the node, and we use a representative snippet, translated through Google translate, to give the gist of the article. As you continue, you can find concepts that are listed in many languages - but you can still see how the same concept might be described differently.
We've done some evaluations on the system...
Some tools we've developed for researchers: an API, WikAPIdia and WikiBrain (get access from GitHub). These are tools that download and organize Wikipedia datasets. You can do eg. relatedness metrics (eg., how related 'engine' and 'car' are in different languages).
Some comments & questions - there are huge difference in the practices around different language editions - first-mover advantages, management of articles, etc.
What we were doing (in part) was a pushback against a lot of data scientists and AI researchers who use the English version of Wikipedia as a baseline. The degree and extent to which there is unique content across languages is shocking - only 1 percent conceptual overlap over 25 languages.
Comment - if there ar diverse concepts, then the philosophy of a 'neutral point of view' is contrary to fact. They (Wikipedians) believe there is a world-wide 'neutral point of view'.
Mass-Collaboration as a basis for procedures for e-participation
Thomas Herrman
How can several people be supported to contribute to the solving of problems in the social context? In this situation, how is creativity encouraged? How can democratic principles be taken into account? How large a group can this support?
The concepts of social and collaborative creativity partially overlap; so do the concepts of mass collaboration, e-participation, and social/cultural creativity (but there is 0 overlap in Google Scholar for all three).
In Germany, there are laws supporting co-determination that determines the degree of co-management and employee contribution (mitbestimmung). There are special teams called Betriebrat that manage this. So eg. there are discussions concerning working conditions, layoff, which ultimately allow for co-determination among people.
There is a range of increasing degrees of participation, from informing people and observing people to interaction to co-management of outcomes. These have technical equivalents ranging from linear MOOCs to big data to brainstorming through to discussion forums, open source and discussion and e-voting.
So now: the case of the German Citizen dialogue on demographic trends. The goal was to propose improvements to society. It took place in cities, with 80 people each. Three major themes:
- how will we live together
- life-long learning
- the influence of this on work environments
There people sitting at individual tables (10 tables of 8?) which would address one of these questions for th full day, and report back. There were two phases: first, the comment on the current situation, and second, proposals for improvements.
Afterword, representatives from the cities were called to meet in Berlin, and a final report was produced.
Facilitation: we had at every table a facilitator and a taker of minutes. Participants didn't see the notes being taken; from time to time a summary was read to the participants. Results were not visualized; the goals were not visible. Facilitators tried - with limited success - to encourage less active participants.
Role of the experts: experts were invited to give an opinion, but then citizens complained that they did not want to be the victim of expert participation, so in the next round the experts stayed in the background and waited for the citizens to ask for help, which never happened. Experts were not asked to support the finding or proposals or to clarify whether an idea had been tested elsewhere. There was a strong focus on practical regional knowledge, but this was not compared at the global level of what has already been done. A lot of singular experiences were discussed but there was little attempt to discuss at a systemic level.
Overall, there was a low level of interaction between the tables - sometimes highlights were presented (by the opinion leaders) and participants had no input in the production of the final report. The experts made additional contributions which were included without any critical discussion.
Lessons learned: people who are highly interested and willing to be engaged are not necessarily well-prepared to contribute novel ideas. The process of converging a huge number of contributions and exploit potential for synergy is difficult. And the influence people have on real political decisions was unclear - was it really about mass collaboration, or merely mass contributions.
So the questions are: how can people be encouraged to relate their ideas to each other? Eg. to escape the hidden profile trap - people take out of the whole set of mass contributions those parts which sound familiar to them. The group does not base its decisions on the information because it's not really shared. It's difficult for people to relate to what others have said.
Also, how can participants be motivated to take existing knowledge or expertise into account?
How can the dominator-follower relation be transformed into a symmetrical relation?
And how can research on small group creativity support be transformed to the level of large numbers of participants? How can the transition from mass contributing to mass collaboration be defined?
Which facilitation studies are sufficient - visualization, prompting, etc.?
Why do people takee part in mass collaboration? Many of them just like it - it's not because they believe they will create change, it's just because they like the experience of participation.
There is a need for scaffolding and prompting, to support directly referencing to others' experiences - alternating between contributing and comparison of contributions, detecting the most interesting similarities and contrasts, to create something 'new'. Needed, the activation of more passive people.
We look for socio-technical approaches to maintain the awareness of existing information - maybe we need several facilitators, to represent different positions or interest groups (eg. if you have a conservative facilitator, he or she will filter out more progressive contributions). The political directions of the attendees should be mirrored at the facilitator level.
In general, we may have:
- type A1 - not appropriate to be carried out in small groups
- type A2 - not appropriate to be carried out via mass collaboration
and
- type B1 - more efficient when carried out in small groups,
- type B2 - more efficient when carried out vie mass collaborations
How can sociotechnical solutions support a shift from A to B? Example, production-blocking and fear of evaluation in small groups can be avoided by organizational and technical measures, eg., anonymous comments submitted electronically.
Claim: facilitation is needed!? Question: can facilitation develop spontaneously? The whole facilitation business is developed on the basis of the ineffectiveness and inability of small groups.
What is facilitation? 'any activity that makes tasks for others easy'.
A procedure is proposed:
- preparing participants (mess finding, data finding, etc) - often we get brainstorming, etc., without this phase of preparation
- making contributions - online we typically think of written contributions, maybe there's a way to do oral contributions (it's easier for a lot of people) - at the same time taking new content into account
- observe the process of merging
So we need to shift facilitation work to support mass participation.
- from one facilitator, to several facilitators (plus a metafacilitator)
- from qualitative comparison of viewpoints, to quantitative evaluations
- from., giving every contribution equal weight, to, the main perspectives should be represented
- from simple voting mechanisms, to complex voting mechanisms
- from contributions visible to all participants, to extra support needed to make the facilitators work visible
- from interventions and prompts being delivered to all, to them being delivered selectively
The transition from collaboration to mass collaboration is not clear. Mechanisms arre needed to build the most promising subsets (7 +/- 2). Collaborative facilitation is needed for prompts and representing diverging positions.
Comments: differences between small groups and mass - eg. scientific community as an example of mass collaboration. Journal editors as facilitators? It seems odd - the facilitators are mostly for co-located groups. In mass collaboration, facilitation is created by means of the structures. Eg. Wikipedia has a particular set of structures and norms, Linus does, web communities do, but they typically don't have facilitators. (But - by contrast, go back to the field of business in the 1950s - that was mass collaboration - why shouldn't the same improvement be possible in mass collaboration - it is worth at least typing, to not only rely on the structures).
Comment: comparing the role of curation and facilitation. Even in the OS community, people identify needs. In this context we have virtual ecologies of participation. The facilitator role is changing. Response:there is less research in the area of the effects of facilitation in the context of mass collaborations.
(SD - the presumption here seems to be that people are not participating correctly - but maybe that's backward)
Comment: people are typing to scale from small groups to masses; and on the other side, people who study masses, less interested in impacting or steering or moving the masses. Also there is this concept of 'liquid democracy' by means of technology. Maybe there is research on this.
Summary Discussion
Major issues:
- collaboration vs cooperation
- democracy: governance vs democracy, collaboration vs right of minorities
- how masses really operate
- can we design or influence how masses think or operate - can we make the masses more effective? the MOOC?
- (?) tools to study mass collaboration
Discussion: concept of the 'flipped classroom' - we do our readings ahead of time - somee of us did extended abstracts - but these create resources constraints. We see this with MOOCs - they are not a big success, because they require additional resource commitments (on the part of students). And similarly with aa book - it creates resource constraints.
Moving beyond the workshop? (Discussion of the idea of publication)
Understanding and Bridging the Wikipedia Language Gap
There is an extensive literature on Wikipedia, everything from collaboration to participation to embedded bias to the uses of Wikipedia data structures by AI and natural language processing application.
So, in different places people speak different languages, but the presumption is people are reporting on the same things, and have the same sort of coverage biases. But as we consider the gloabl nature of information we need to think about equitable information access, retaining diverse perspectives, and algorithmic biases and the role they play in information structure, representation, and consumption. A part of this talk is to look at tools that address this.
Gerhard Fisher presented a nice picture of how things are - this presentation is more about how things should be.
To what extent is there content diversity among Wikipedia language editions? I thought this would be a simple question - five years ago. There are two aspects here in the implicit assumption that every language describes the same sorts of things, and so there is a global consensus among the concepts. But there is diversity at the concept and sub-concept levels.
We have concept diversity - that is, the set of titles of articles.
We have sub-concept diversity - that is, the set of topics within a given article.
First of all, the methodology, matching pages to concepts. So we want (eg.) to find a core concept ('chocolate') and then match the pages in different languages to this topic. There are 'language links' on the wikipedia pages that will take you to the other language for the same page. We take a hyperlingual approach, which gives us a cluster of pages (sometimes even if the link between two language editions is missing).
The 'conceptual drift' problem also occurs - the boundaries of conceptions changes across language. So, eg., we have a page in English on 'river', which links to a German page, which links bacl to 'canal', which links to a French page, which links back to 'canyon'. So the algorithm has parameters to tune the tightness of concepts - it first limits the number of edges from a given language, and second, has a minimum number of languages to allow an edge to remain.
So after examining the concepts we discovered that the archetypical concept looks nothing like 'chocolate'. They look more like a region or place or person (explore this?) We also find that the bulk of concepts are single-language concepts, with only a small number of concepts common to even three languages.
So - why? Maybe it's just that English, being the largest, is a superset. And everything else is just growing into it. But when we do pairwise comparisons this doesn't bear out - comparing English to German, for example, we find 40 percent of the concepts unique to each language.
So - analyzing the concepts - we can (Adafre and de Rijke, 2006) use the links on the page to create a conceptual representation of the page. Now we can compare these between languages. So we look at which links they have in common, which defines an 'overlap coefficient'. The mean OC is 0.41 - that is, about 59 percent of the links in the shorter articles (in one language) don't appear in the larger articles (in another language).
So, what accounts for this diversity?
We look at the concept of self-focus - the degree to which language-based cultures centre their descriptions around objects and entities of regional importance. So we used geo-tags to examine this, drawing out articles with a spacial location. We use a technique called 'spatial indegree sums' - so, for example, we take an area like 'Poland', find the entities within that region, and then find all the articles that link to those original entities. So we can give a particular weight to a given geographic area. This allows us to compare an indegree sum for different languages for a given geographic area, which allows us to measure self-focus.
So - the global consensus hypothesis would suppose that each region has relatively equal indegree sums for each region, while the diversity hypothesis would predict that the polish-language Wikipedia would favour Poland. The actual results show regional diversity - people regionalize the information around them. The English Wikipedia focuses on the US and Britain, the Russian Wikipedia on Russia, etc.
So: self-focus is a systemic boas in Wikipedia. people orient the knowledge and language around themselves. This diversity is a serious concern for semantically based AI and NLP applications.
On the converse side: we could create applications that take advantage of this diversity. We built a system called Omnipedia that provides access to 85M+ concepts in 25 languages. Pick a concept - it displays the number of related links for each of the languages, as nodes: you click on the node, and we use a representative snippet, translated through Google translate, to give the gist of the article. As you continue, you can find concepts that are listed in many languages - but you can still see how the same concept might be described differently.
We've done some evaluations on the system...
Some tools we've developed for researchers: an API, WikAPIdia and WikiBrain (get access from GitHub). These are tools that download and organize Wikipedia datasets. You can do eg. relatedness metrics (eg., how related 'engine' and 'car' are in different languages).
Some comments & questions - there are huge difference in the practices around different language editions - first-mover advantages, management of articles, etc.
What we were doing (in part) was a pushback against a lot of data scientists and AI researchers who use the English version of Wikipedia as a baseline. The degree and extent to which there is unique content across languages is shocking - only 1 percent conceptual overlap over 25 languages.
Comment - if there ar diverse concepts, then the philosophy of a 'neutral point of view' is contrary to fact. They (Wikipedians) believe there is a world-wide 'neutral point of view'.
Mass-Collaboration as a basis for procedures for e-participation
Thomas Herrman
How can several people be supported to contribute to the solving of problems in the social context? In this situation, how is creativity encouraged? How can democratic principles be taken into account? How large a group can this support?
The concepts of social and collaborative creativity partially overlap; so do the concepts of mass collaboration, e-participation, and social/cultural creativity (but there is 0 overlap in Google Scholar for all three).
In Germany, there are laws supporting co-determination that determines the degree of co-management and employee contribution (mitbestimmung). There are special teams called Betriebrat that manage this. So eg. there are discussions concerning working conditions, layoff, which ultimately allow for co-determination among people.
There is a range of increasing degrees of participation, from informing people and observing people to interaction to co-management of outcomes. These have technical equivalents ranging from linear MOOCs to big data to brainstorming through to discussion forums, open source and discussion and e-voting.
So now: the case of the German Citizen dialogue on demographic trends. The goal was to propose improvements to society. It took place in cities, with 80 people each. Three major themes:
- how will we live together
- life-long learning
- the influence of this on work environments
There people sitting at individual tables (10 tables of 8?) which would address one of these questions for th full day, and report back. There were two phases: first, the comment on the current situation, and second, proposals for improvements.
Afterword, representatives from the cities were called to meet in Berlin, and a final report was produced.
Facilitation: we had at every table a facilitator and a taker of minutes. Participants didn't see the notes being taken; from time to time a summary was read to the participants. Results were not visualized; the goals were not visible. Facilitators tried - with limited success - to encourage less active participants.
Role of the experts: experts were invited to give an opinion, but then citizens complained that they did not want to be the victim of expert participation, so in the next round the experts stayed in the background and waited for the citizens to ask for help, which never happened. Experts were not asked to support the finding or proposals or to clarify whether an idea had been tested elsewhere. There was a strong focus on practical regional knowledge, but this was not compared at the global level of what has already been done. A lot of singular experiences were discussed but there was little attempt to discuss at a systemic level.
Overall, there was a low level of interaction between the tables - sometimes highlights were presented (by the opinion leaders) and participants had no input in the production of the final report. The experts made additional contributions which were included without any critical discussion.
Lessons learned: people who are highly interested and willing to be engaged are not necessarily well-prepared to contribute novel ideas. The process of converging a huge number of contributions and exploit potential for synergy is difficult. And the influence people have on real political decisions was unclear - was it really about mass collaboration, or merely mass contributions.
So the questions are: how can people be encouraged to relate their ideas to each other? Eg. to escape the hidden profile trap - people take out of the whole set of mass contributions those parts which sound familiar to them. The group does not base its decisions on the information because it's not really shared. It's difficult for people to relate to what others have said.
Also, how can participants be motivated to take existing knowledge or expertise into account?
How can the dominator-follower relation be transformed into a symmetrical relation?
And how can research on small group creativity support be transformed to the level of large numbers of participants? How can the transition from mass contributing to mass collaboration be defined?
Which facilitation studies are sufficient - visualization, prompting, etc.?
Why do people takee part in mass collaboration? Many of them just like it - it's not because they believe they will create change, it's just because they like the experience of participation.
There is a need for scaffolding and prompting, to support directly referencing to others' experiences - alternating between contributing and comparison of contributions, detecting the most interesting similarities and contrasts, to create something 'new'. Needed, the activation of more passive people.
We look for socio-technical approaches to maintain the awareness of existing information - maybe we need several facilitators, to represent different positions or interest groups (eg. if you have a conservative facilitator, he or she will filter out more progressive contributions). The political directions of the attendees should be mirrored at the facilitator level.
In general, we may have:
- type A1 - not appropriate to be carried out in small groups
- type A2 - not appropriate to be carried out via mass collaboration
and
- type B1 - more efficient when carried out in small groups,
- type B2 - more efficient when carried out vie mass collaborations
How can sociotechnical solutions support a shift from A to B? Example, production-blocking and fear of evaluation in small groups can be avoided by organizational and technical measures, eg., anonymous comments submitted electronically.
Claim: facilitation is needed!? Question: can facilitation develop spontaneously? The whole facilitation business is developed on the basis of the ineffectiveness and inability of small groups.
What is facilitation? 'any activity that makes tasks for others easy'.
A procedure is proposed:
- preparing participants (mess finding, data finding, etc) - often we get brainstorming, etc., without this phase of preparation
- making contributions - online we typically think of written contributions, maybe there's a way to do oral contributions (it's easier for a lot of people) - at the same time taking new content into account
- observe the process of merging
So we need to shift facilitation work to support mass participation.
- from one facilitator, to several facilitators (plus a metafacilitator)
- from qualitative comparison of viewpoints, to quantitative evaluations
- from., giving every contribution equal weight, to, the main perspectives should be represented
- from simple voting mechanisms, to complex voting mechanisms
- from contributions visible to all participants, to extra support needed to make the facilitators work visible
- from interventions and prompts being delivered to all, to them being delivered selectively
The transition from collaboration to mass collaboration is not clear. Mechanisms arre needed to build the most promising subsets (7 +/- 2). Collaborative facilitation is needed for prompts and representing diverging positions.
Comments: differences between small groups and mass - eg. scientific community as an example of mass collaboration. Journal editors as facilitators? It seems odd - the facilitators are mostly for co-located groups. In mass collaboration, facilitation is created by means of the structures. Eg. Wikipedia has a particular set of structures and norms, Linus does, web communities do, but they typically don't have facilitators. (But - by contrast, go back to the field of business in the 1950s - that was mass collaboration - why shouldn't the same improvement be possible in mass collaboration - it is worth at least typing, to not only rely on the structures).
Comment: comparing the role of curation and facilitation. Even in the OS community, people identify needs. In this context we have virtual ecologies of participation. The facilitator role is changing. Response:there is less research in the area of the effects of facilitation in the context of mass collaborations.
(SD - the presumption here seems to be that people are not participating correctly - but maybe that's backward)
Comment: people are typing to scale from small groups to masses; and on the other side, people who study masses, less interested in impacting or steering or moving the masses. Also there is this concept of 'liquid democracy' by means of technology. Maybe there is research on this.
Summary Discussion
Major issues:
- collaboration vs cooperation
- democracy: governance vs democracy, collaboration vs right of minorities
- how masses really operate
- can we design or influence how masses think or operate - can we make the masses more effective? the MOOC?
- (?) tools to study mass collaboration
Discussion: concept of the 'flipped classroom' - we do our readings ahead of time - somee of us did extended abstracts - but these create resources constraints. We see this with MOOCs - they are not a big success, because they require additional resource commitments (on the part of students). And similarly with aa book - it creates resource constraints.
Moving beyond the workshop? (Discussion of the idea of publication)
Comments
Post a Comment
Your comments will be moderated. Sorry, but it's not a nice world out there.