#!/usr/bin/perl -w

=head1 NAME

yatea - Perl script for extracting terms from a corpus of texts and
providing a syntactic analysis in a head-modifier representation.

=head1 SYNOPSIS

yatea [-help] [-man] [--rcfile=file] file

=head1 OPTIONS

=over 4

=item    B<--help>            brief help message

=item    B<--man>             full documentation

=item    B<--rcfile=file>     load the given configuration file

=item    I<file>              corpus of texts in Flemm or TreeTagger output format


=back

=head1 DESCRIPTION

YaTeA aims at extracting noun phrases that look like terms from a
corpus. It also provides their syntactic analysis in a head-modifier
format.

As input, the term extractor requires a corpus which has been
segmented into words and sentences, lemmatized and tagged with
part-of-speech (POS) information. The input file is encoded in
UTF-8. Currently, the text has to be postagged by TreeTagger (for
French and English), or TreeTagger+Flemm for French.

As output, the script makes a directory containing the results in
various formats (according the configuration): XML, text and
TreeTgger-like output.

==head2 USE OF YATEA

Using YaTeA requires to have a output of TreeTagger
(<http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/DecisionTreeTagger.html>. It
will be the input of YaTeA.

To run yatea, a configuration file is needed (usually yatea.rc in
/etc/yatea). This file describes the behaviour of the term extractor
for French and English texts. You have to indicate the language of the
configuration file you use (see section CONFIGURATION FILE FORMAT for
more details, below). It also indicates the path of the configuration files
for the linguistic analysis. You have to adapt the path if your
configuration is not standard.

An example of the configuration file is available in
C<etc/yatea/yatea.rc> from the archive directory.

=over 4

=item The most common command line to run YaTeA is 

C<yatea TreeTaggerOutputFile.ttg>

It is assumed that the directory containing the program yatea is in
your PATH variable and that the configuration file is C</etc/yatea/yatea.rc>.

=item If you are not allow to copy the configuration file C<yatea.rc>
in the directory C</etc/yatea> (or create this directory), or if you
want to use your own configuration file, you can specify the file with its path by using the option C<--rcfile>

C<yatea --rcfile MyYaTeAConfig.rc TreeTaggerOutputFile.ttg>

=back

=head2 EXPLANATION ON THE ANALYSIS


The main strategy of analysis of the term candidates is based on the
exploitation of simple parsing patterns and endogenous
disambiguation. Exogenous disambiguation is also made possible for the
identification and the analysis of term candidates by the use of
external resources, I<i.e.> lists of testified terms.

Endogenous disambiguation consists in the exploitation of intermediate
chunking and parsing results for the parsing of a given Maximal Noun
Phrase (MNP). This feature allows the parse of complex noun phrases
using a limited number of simple parsing patterns (80 patterns
containing a maximum of 3 content words in the experiments described
below). All the MNPs corresponding to parsing patterns are parsed
first. In a second step, remaining unparsed MNPs are processed using
the results of the first step as I<islands of reliability>.  An
I<island of reliability> is a subsequence (contiguous or not) of a MNP
that corresponds to a shorter term candidate that was parsed during
the first step of the parsing process. This subsequence along with its
internal analysis is used as an anchor in the parsing of the
MNP. Islands are used to simplify the POS sequence of the MNP for
which no parsing pattern was found. The subsequence covered by the
island is reduced to its syntactic head. In addition, islands increase
the degree of reliability of the parse. When no resource is provided
and as there is no parsing pattern defined for the complete POS
sequence "NN NN NN of NN" corresponding to the term candidate
"Northern blot analysis of cwlH", the progressive method is
applied. In such a case, the TC is bracketed from the right to the
left, which results in a poor quality analysis. When considering the
island of reliability "northern blot analysis", the correct bracketing
is found.

=head1 INPUT/OUTPUT FILE FORMATS

=head2 Yatea input: TreeTagger output 

TODO

-head2 YaTeA XML output

TODO

=head1 CONFIGURATION FILE FORMAT

=head2 Main configuration file (usually C<yatea.rc>)

The configuration file of YaTeA is divided into two sections:

=over 

=item * Section C<DefaultConfig>

=over

=item * 

C<CONFIG_DIR> : directory containing the configuration files according to the language

=item * 

C<LOCALE_DIR> : directory containing the environment files according to the language

=item * 

C<RESULT_DIR> : directory where are stored the results (probably not useful)

=back

=item * Section C<OPTIONS>

=over

=item * 

C<language> I<language> : Definition of the language of the
corpus. Values are either C<FR> (French - TreeTagger output - TagSet
<http://www.ims.uni-stuttgart.de/~schmid/french-tagset.html>),
C<FR-Flemm> (French - output of Flemm analyser or C<EN> (English -
TreeTagger or GeniaTagger output - PennTreeBank Tagset)

=item * 

C<suffix> I<suffix> : Specification of a name for the current version
of the analysis. Results are gathered in a specific directory of this
name and result files also carry this suffix

=item *

C<output-path> : set the path to the directory that will contain the
 results for the current corpus (default: working directory)

=item * 

C<termino> I<File> : Name of a file containing a list of
testified terms. 

=item * 

C<monolexical-all> : all occurrences of monolexical phrases
are considered as term candidates. The value is 0 or 1.

=item * 

C<monolexical-included> : occurrences of monolexical term
candidates that appear in complex term candidates are also displayed. The value is 0 or 1.

=item * 

C<match-type> [loose or strict] :

=over

=item * 

C<loose> : testified terms match either inflected or lemmatized forms of each word

=item * 

C<strict> : testified terms match the combination of inflected form and POS tag of each word

=item * 

unspecified option: testified terms match match inflected forms of words

=back

=item * 

C<xmlout> : display of the parsed term candidates in XML format. The
value is 0 or 1.

=item * 

C<termList> : display of a list of terms and sub-terms along with
their frequency. To display only term candidates containing more than
one word (multi-word term candidates), specify the value C<multi>.
All term candidates will be displayed , monolexical and multi-word
term candidates with the value C<all>, or if any value is specified.

=item * 

C<printChunking> : displays of the corpus marked with phrases in a
HTML file along with the indication that they are term candidates or
not. The value is 0 or 1.

=item * 

C<TC-for-BioLG> : annotation of the corpus with term candidates in a
XML format compatible with the BioLG software. The value is 0 or 1.

=item * 

C<TT-for-BioLG> : annotation of the corpus with testified terms in a
XML format compatible with the BioLG software. The value is 0 or 1.
(http://www.it.utu.fi/biolg/, biological tuned version of the Link
Grammar Parser)

=item * 

C<XML-corpus-for-BioLG> : creation of a BioLG compatible XML version
of the corpus with PoS tags marked form each word. The value is 0 or 1.

=item * 

C<debug> : displays informations on parsed phrases (i.e. term
candidates) in a text format. The value is 0 or 1.


=item * 

C<annotate-only> : only annotate testified terms (no acquisition). The
value is 0 or 1.

=item * 

C<TTG-style-term-candidates> : term candidates are displayed in
TreeTagger output format. Term separator is the sentence boundary tag
C<SENT>. To extract only term candidates containing more than one
word (multi-word term candidates), specify the option C<multi>. 
All term candidates will be displayed , monolexical and multi-word
term candidates with the value C<all>, or if any value is specified.

=back

=back


=head2 Linguistic configuration files

Available in C<share/YaTeA/config> from the archive directory.

TODO

=head1 EXAMPLES

TODO

=head1 CONTRIBUTORS

=over

=item *

Charlotte Roze has defined the configuration files to process a corpus
tagged with Flemm

=item * 

Wiktoria Golik, Robert Bossy and Claire Nédellec (MIG/INRA) have
corrected bugs and improve the mapping of testified terms.


=back

=head1 SEE ALSO

Sophie Aubin and Thierry Hamon. Improving Term Extraction with
Terminological Resources. In Advances in Natural Language Processing
(5th International Conference on NLP, FinTAL 2006). pages
380-387. Tapio Salakoski, Filip Ginter, Sampo Pyysalo, Tapio Pahikkala
(Eds). August 2006. LNAI 4139.

=head1 AUTHORS

Thierry Hamon <thierry.hamon@univ-paris13.fr> and Sophie Aubin <sophie.aubin@lipn.univ-paris13.fr>

=head1 LICENSE

Copyright (C) 2005 by Thierry Hamon and Sophie Aubin

This program is free software; you can redistribute it and/or modify
it under the same terms as Perl itself, either Perl version 5.8.6 or,
at your option, any later version of Perl 5 you may have available.

=cut



#######################################################################
#
# Last Update: 20/11/2012 (mm/dd/yyyy date format)
# 
# Copyright (C) 2004 Thierry Hamon, Sophie Aubin
#
# Written by thierry.hamon@univ-paris13.fr, sophie.aubin@lipn.univ-paris13.fr
#
# Author : Thierry Hamon - Sophie Aubin
# Email : thierry.hamon@univ-paris13.fr - sophie.aubin@lipn.univ-paris13.fr
# URL : http://www-limbio.smbh.univ-paris13.fr/membres/hamon/
# Version : 1
########################################################################


use strict;
use warnings;

use Getopt::Long;
use Config::General;
use Pod::Usage;
use Data::Dumper;
use Lingua::YaTeA::Corpus;
# use Lingua::YaTeA::Config;
use Lingua::YaTeA;


# Process Option

my $man = 0;
my $help = 0;
my $rcfile = "";

GetOptions('help|?' => \$help, man => \$man, "rcfile=s" => \$rcfile) or pod2usage(2);
pod2usage(1) if $help;
pod2usage(-exitstatus => 0, -verbose => 2) if $man;

my %config = Lingua::YaTeA::load_config($rcfile);

my $current_dir = `pwd`;
my $corpus_path = $ARGV[$#ARGV];

# warn "$corpus_path\n";

    print STDERR  "\n
\n
#     #     #    #########  #######     #
 #   #     # #       #      #          # #
  # #     #   #      #      #         #   #
   #     #     #     #      #####    #     #
   #     #######     #      #        #######
   #     #     #     #      #        #     #
   #     #     #     #      #######  #     #
\n\n";

my $sys_config ;
my $yatea;
my $corpus;

$yatea = Lingua::YaTeA->new($config{"OPTIONS"}, \%config);

if (defined $corpus_path) {
    if (-f $corpus_path)
    {
	$corpus = Lingua::YaTeA::Corpus->new($corpus_path,$yatea->getOptionSet,$yatea->getMessageSet);
    }
    else
    {
	die("\"".$corpus_path . "\"". $yatea->getMessageSet->getMessage("NO_FILE")->getContent($yatea->getOptionSet->getDisplayLanguage) . "\n");
    }
    $yatea->termExtraction($corpus);
} else {
    die( $yatea->getMessageSet->getMessage("NO_FILE_ARG")->getContent($yatea->getOptionSet->getDisplayLanguage) . "\n");
}

# sub usage
# {
#     my ($program_name) = @_;
#     $program_name =~ s/^\s*\.\///;

#      warn "\n
#            ********************************************
#            *                Using " . $program_name . "               *
#            ******************************************** 
# \nCommand : ". $program_name . " fichier.ttg 
# Options :
# \t-help or -h : displays this message
# \t-rcfile <File> : name of the file containing the configuration
# \t-lang <language> : language of the corpus : FR: French ou EN: English
# \t-suffix <suffix> : specification of a name for the current version of the analysis. Results are gathered in a specific directory of this name and result files also carry this suffix
# \t-termino <File> : name of a file containing a list of testified terms. To specify several files, repeat the -termino switch for each
# \t-monolexical-all : all occurrences of monolexical phrases are considered as term candidates
# \t-monolexical-included : occurrences of monolexical term candidates that appear in complex term candidates are also displayed
# \t-match-type [\"loose\" or \"strict\"]: \n\t\t-loose: testified terms match either inflected or lemmatized forms of each word\n\t\t-strict: testified terms match the combination of inflected form and POS tag of each word\n\t\t-unspecified option: testified terms match match inflected forms of words

# Options for result display:
# \t-xmlout :  display of the parsed term candidates in XML format
# \t-termList : display of a list of terms and sub-terms along with their frequency. To extract only term candidates containing more than one word (multi-word term candidates), specify the option \"multi\". By default, all term candidates will be extracted, monolexical and multi-word term candidates
# \t-printChunking : displays of the corpus marked with phrases in a HTML file along with the indication that they are term candidates or not 
# \t-TC-for-BioLG : annotation of the corpus with term candidates in a XML format compatible with the \"biolg\" version of the Link Grammar Parse
# \t-TT-for-BioLG : annotation of the corpus with testified terms in a XML format compatible with the \"biolg\" version of the Link Grammar Parser
# \t-XML-corpus-for-BioLG : creation of a BioLG compatible XML version of the corpus with PoS tags marked form each word
# \t-debug : displays informations on parsed phrases (i.e. term candidates) in a text format
# \t-TTG-style-term-candidates : term candidates are displayed in TreeTagger output format. Term separator is the sentence boundary tag \"SENT\". To extract only term candidates containing more than one word (multi-word term candidates), specify the option \"multi\". By default, all term candidates will be extracted, monolexical and multi-word term candidates

#    \n";
#     die "\n";
# }

# TODO (?):

# (OUI)\t-correction: allows the correction of POS or lemma information of the MNPs according to the testified terms provided


# (OUI)\t-basicTerms: display of the list of basic term candidates (every pairs of words in a Head-Modifier relationship)
# (NON?)\t-MNPhtml : display of the corpus with the occurrences of the MNPs marked in a HTML file. Information on each MNP is provided. 
# (NON?)\t-compareChunking <File>: comparison of the current chunking of the corpus with a previous version specified as <File> (CORPUS_Chunking_corpus.html)
# (OUI) \t-annotate : only annotate testified terms (no acquisition)

__END__


