CLARIN AAI Vision. Daan Broeder Max-Planck Institute for Psycholinguistics. DFN meeting June 7 th Berlin

Transcription

1 CLARIN AAI Vision Daan Broeder Max-Planck Institute for Psycholinguistics DFN meeting June 7 th Berlin

2 Contents What is the CLARIN Project What are Language Resources A Holy Grail CLARIN User Scenario AAI Vision and what needs to be solved to achieve it

3 What is CLARIN Common Language Resources and Technology Infrastructure The CLARIN project is a large-scale pan- European collaborative effort to coordinate and make language resources and technology available and readily useable for Language & SSH (Social Sciences & Humanities) researchers.

4 Language Resources Any resource used to study language Text Corpora Newspapers,, , sms messages Multi-media corpora Audio recordings to study phonetics, train speech recognizers Video recordings for Sign-Language studies Language Documentation (language use in cultural context) Multi-Media Lexica Lexical entries linked with pictures, sound

5 Sign-Language Example

6 Multi-Media lexicon example Lexical entries link directly into archived corpora, e.g. via Annex

7 What is CLARIN CLARIN is an EU Infrastructure project with 4.2 ME funding for a 3 year preparatory phase (ends 2010) Additional funding from national governments (at this moment at least 14 ME ) The CLARIN consortium has now 36 partners from 26 EU countries The CLARIN community has >180 member organisations in 32 countries (mostly from NLP orgs.) CLARIN is based on many earlier initiatives with many participants: LangWeb, EARL, TELRI, LIRICS and more recent DAM-LR MPI for Psycholinguistics is responsible for WP2; working on the technical infrastructure

8 CLARIN Time Plan Preparatory Phase Limited set of federated CLARIN centers (10+) Showcases, demonstrators WP8 Investigation national funding for the construction & maintenance phase Construction Phase No direct European funding but EU assisting projects Depend on national project commitments Netherlands already until 2014 currently intensive preparations for CLARIN D (->2016) 2016? - Operational Phase Has to be cost efficient, we have to compete! CLARIN EU continuation after the preparatory phase is likely in the form of an ERIC important if only to provide a legal entity to make contracts with outside parties on behalf of the CLARIN community.

9 A backbone of CLARIN centers These together uphold the infrastructure, maintaining it and offer guidance & expertise for its use. Have stable repositories for resources and services Need strong national support for many years Need good teams that have a long time perspective and can provide persistency and continuation of knowledge This is yet far from reality Current situation is one of accidental and temporary collaborations and obligations Only a limited number of centers can probably fulfill the criteria of sufficient stability, funding and technological strength Currently 25 candidate centers

10 CLARIN Holy Grail User Scenario A researcher authenticates at his own organization and creates a virtual collection of resources from different repositories. He does this on the basis of browsing a catalogue, searching through metadata, or searching in resource content. To be granted access to this distributed dataset he signs the appropriate licenses He is then able to use a workflow specification tool and process this virtual collection using LT tools in the form of reliable distributed web services which he is authorized to use. (Intermediate) results are stored in a user specific workspace After evaluation, the resulting data (including metadata) can be added to a repository and the virtual collection specification can be stored for future reference For our domain this is ambitious and challenging, but even a partial realization is worthwhile

11 CLARIN Infrastructure Components In the previous scenario we find the following components & functionality Metadata catalog Virtual collection registries Persistent Identification of Resources EPIC: European PID Consortium: GWDG, CSC, SARA AAI infrastructure Technical issues Organizational Legal

12 Virtual Language Observatory CGN (12.000) OAI PMH harvesting IMDI Domain End.Lang. (35.000) MPI (33.000) and transformation GIS overlay BAS (7.400) AILLA (1.800) OLAC (40.000) Indexes Facetted Browser LRT Inventory (800/137) DFKI Tool Registry (292) ELDA (60) others hard problem: - mapping - granularity - curation Catalogue

13 CLARIN AAI Purpose is to create one single domain of CLARIN resources and services for our users Where users have only one identity (and since we hope to have very many users) preferably maintained at their home institute and can use SSO (single sign on) between the centers Our users are linguists and SSH academics spread out over Europe, CLARIN can not hope to influence the way their user accounts are set-up. But CLARIN can profit from existing AAI systems in the research & education domain. CLARIN centers are part of the CLARIN organization and they can be asked to conform to specific standards wrt. AAI

14 Federated Authentication Many countries have a National Identity Federation (IDF) set up by the different NRENs (national research education network) Such a federation is a collection of IdPs and SP Users have an account at their institute (IdP) and can use resources or services from centers (SPs) When a user accesses a resource at a SP he can authenticate at his own IdP 1 IdP 2 3 SPa 5 resources SPb user Purpose: info Provide SSO Single user identity Limited user information exposure 4,6 processing resources

15 CLARIN wide AAI (1) The CLARIN SPs become members of their national IDFs Rely on the edugain confederation (GEANT 3 project) to provide the trust between the national IdFs IDF a SP1 edugain is not yet functional attribute harmonization issues privacy issues disclosing attributes when crossing national frontiers SP2 edugain Metadata & trust IDF b homeless users? IDF c SP3

16 CLARIN wide AAI (2) Establish a CLARIN SP organization as a legal entity able to sign contracts where needed with the national IDFs CLARIN SP organization takes care of exchanging the SP specifications with the national IDFs IDF a SP1 Metadata & trust SP2 SP3 IDF b homeless users? IDF c

17 How about licenses? Many resources are available under a special license (EULA) e.g. Academic use only CLARIN WP7 investigated possible harmonization Should a user have to repeatedly sign the same EULA at different data provider when processing a distributed data set? This would break the SSO! Can we store the signed EULA information at the users IdP as an attribute? CLARIN has no way of influencing the IdP organizations so a CLARIN registry for this would be needed

18 Virtual Organization Platform user browser SPa SPb VO Platform External User Attribute Authority There is a PoC implementation available This is suitable as a basis for a CLARIN EULA service. Developing this further (probably) part of CLARIN NL IdP EULA DB Create special EULA service. This is part of the CLARIN organization independent of the IDFs

19 CLARIN SP Test Federation The national Identity Federations (IDF) will come together in a single confederation: edugain This way users associated with any IdP can use resources from any SP in the confederation This is not operational yet Therefore CLARIN created a SP federation that can sign contracts with the individual IDFs This is an administrative burden but: it works!, is extendible and independent of edugain progress Current status Initial Service Provider Federation: MPI-Psyl, BBAW, IDS, CSC Made contract with HAKA Finland, DFN AAI Germany, SURFfed Netherlands Successfully demonstrated SSO with a few SPs

20 Problems encountered Federation fees for SPs SURFfed, HAKA require payment from external SPs to enter the IDF. All foreign SPs could be considered external. Particular IDF requirements Specific X509 certificate issuer(s) (HAKA) IdP initiated SP connection request (SURFFed) Explaining the SP federation model to all participants SP, IDF management and legal people Scalability of the contracts Important flexibility to add new SPs or national identity federations without too much overhead. One representative for the SPs with power of attorney to deal with the national identity federation agreements (1xN instead of NxN signatures). Currently a CLARIN centre, in the future the CLARIN ERIC

21 National IDF policy What can national IdFs do to make (CLARIN) life easy. Facilitate/push edugain, that would solve most of our problems. Think of harmonizing your contracts (saves the number of annexes in the CLARIN SP contract) Be flexible, be aware of different situations for SPs from other countries e.g.the certificate issuer requirement Don t start asking money for connecting the CLARIN SP federation. We are not commercial publishers Keep cooperating with us, it is going well!

22 Non-EU collaborations Regional Archives Initiative: Cooperation of MPI-Psyl with other organizations interested in EL archiving They use MPI s LAT archiving software Encourage local resource collecting & archiving Network of South American archives has been established and contacts with CLARA were made

23 Non-EU collaborations data sync How will we accommodate users and SPs from non-eu countries? Will we have to wait for a super edugain or can we introduce non-eu IdPs & SPs in the CLARIN federation? Regional Archives Initiative: Cooperation of MPI-Psyl with other organizations interested in EL archiving They use MPI s LAT archiving software Encourage local resource collecting & archiving Network of South American archives has been established

24 collaborations/interactions concrete plans joint projects cooperations contribution PARADE discussions

25 Thank you for your attention CLARIN has received funding from the European Community's Seventh Framework Programme under grant agreement n