Tch. Asst., Faculty of Arts, University of Ljubljana, Aškerčeva 2, SI-1000 Ljubljana, Slovenia; Faculty of Computer and Information Science, University of Ljubljana, Večna pot 113, SI-1000 Ljubljana, Slovenia, luka.tercon@ff.uni-lj.si; ORCID: 0009-0006-3237-3583
PhD, Res. Assoc., Faculty of Arts, University of Ljubljana, Aškerčeva 2, SI-1000 Ljubljana, Slovenia; Jožef Stefan Institute, Jamova 39, SI-1000 Ljubljana, kaja.dobrovoljc@ff.uni-lj.si; ORCID:
0000-0002-5909-7965
PhD, Sr. Res. Assoc., Jožef Stefan Institute, Jamova 39, SI-1000 Ljubljana, Slovenia; Faculty of Computer and Information Science, University of Ljubljana, Večna pot 113, SI-1000 Ljubljana, Slovenia; Institute of Contemporary History, Privoz 11, SI-1000 Ljubljana, Slovenia,
nikola.ljubesic@ijs.si; ORCID:
0000-0001-7169-9152
V članku predstavljamo orodje
CLASSLA-Stanza, cevovod za avtomatsko jezikovno označevanje južnoslovanskih
jezikov, ki temelji na cevovodu za procesiranje naravnega jezika Stanza. Opišemo
vse glavne izboljšave, ki jih prinaša CLASSLA-Stanza v primerjavi s Stanzo in
podamo podroben opis postopka učenja modelov v različici 2.2, najnovejši različici
orodja. Obenem poročamo o rezultatih delovanja cevovoda za različne jezike in
jezikovne zvrsti. CLASSLA-Stanza dosega konsistentno visoke rezultate za vse
podprte jezike in preseže rezultate izvornega cevovoda Stanza pri vseh podprtih
jezikih. Predstavimo tudi novo funkcijo cevovoda, ki omogoča učinkovito
procesiranje spletnih besedil, in opišemo učinkovitost cevovoda za označevanje
transkriptov govora.
Ključne besede: južnoslovanski jeziki,
avtomatsko procesiranje jezika, označevalni cevovod, jezikovno
označevanje
We present CLASSLA-Stanza, a pipeline
for automatic linguistic annotation of South Slavic languages, which is based on
the Stanza natural language processing pipeline. We describe the main improvements
in CLASSLA-Stanza with respect to Stanza and give a detailed description of the
model training process for the latest 2.2 release of the pipeline. We also report
performance scores produced by the pipeline for different languages and language
varieties. CLASSLA-Stanza exhibits consistently high performance across all the
supported languages and outperforms its parent pipeline Stanza at all the
supported tasks. We also present the pipeline’s new functionality that enables
efficient processing of web data and describe the efficiency of the pipeline for
annotating written transcripts of spoken data.
Keywords: South Slavic languages,
automatic linguistic processing, annotation pipeline, linguistic
annotation
The South Slavic languages make up one of the three
major branches of the Slavic language family. Despite being used by around 30 million
people worldwide,Collaboration and Computing for Under-Resourced Languages
(CCURL) workshop, 2016, http://www.lrec-conf.org/proceedings/lrec2016/workshops/LREC2016Workshop-CCURL2016_Proceedings.pdf#page=74.
Although much additional work is required before South
Slavic languages can approach the level of support enjoyed by linguistic giants such
as English, steps have been taken towards establishing common platforms for
supporting the development of new resources and tools for these languages. The CLARIN
Knowledge Centre for South Slavic Languages (CLASSLA)CLASSLA: Knowledge centre for South Slavic languages, https://www.clarin.si/info/k-centre/.CLARIN. The Infrastructure for Language Resources (De
Gruyter, 2022), 429–56.GitHub - clarinsi/classla: CLASSLA Fork of the Official Stanford
NLP Python Library for Many Human Languages, https://github.com/clarinsi/classla/.th Annual Meeting of the Association for Computational Linguistics:
System Demonstrations, 2020, https://doi.org/10.18653/v1/2020.acl-demos.14. 7th Workshop on Balto-Slavic Natural Language
Processing, 2019, https://doi.org/10.18653/v1/W19-3704. Kaja Dobrovoljc et al., “Improving
UD processing via satellite resources for morphology,” paper presented at the Third Workshop on Universal Dependencies (UDW, SyntaxFest
2019), 2019, https://doi.org/10.18653/v1/W19-8004.
The aim of this paper is to provide both a systematic overview of the differences that CLASSLA-Stanza has to the official Stanza pipeline and a description of the model training procedure which was adopted when training models for the latest releases. The description of the training procedure is intended to serve as the main reference for future releases as well as for anyone using the CLASSA-Stanza tool to produce their own models for linguistic annotation.
In accordance with this aim, we first describe the differences between CLASSLA-Stanza and Stanza in Section 2. Section 3 then introduces the datasets used for training the most recent models. Section 4 gives a general description of the model training process, which is followed by an analysis of the results produced by the latest models in Section 5.
At present, the CLASSLA-Stanza annotation tool supports
a total of six tasks: tokenization, morphosyntactic annotation, lemmatization,
dependency parsing, semantic role labelling, and named-entity recognition.
Tokenization is handled by one of two external rule-based tokenizers included in
CLASSLA-Stanza, either the Obeliks tokenizer for standard SlovenianEighth Language Technologies
Conference, 2012, https://nl.ijs.si/isjt12/proceedings/isjt2012_17.pdf. The repository for
the tool can be found at: https://github.com/clarinsi/obeliks. th Workshop on Balto-Slavic Natural
Language Processing, 2015, https://aclanthology.org/W15-5306/. The repository for the tool can be
found at: https://github.com/clarinsi/reldi-tokeniser.
The current version of the models was trained on data
annotated according to three separate systems for morphosyntactic annotation: the
Universal part-of-speech tags and the Universal morphosyntactic features tags, which
are both part of the Universal Dependencies framework for grammatical annotationComputational Linguistics 47, No. 2 (07
2021): 255–308, https://doi.org/10.1162/coli_a_00402. Language Resources and Evaluation 46, No. 1
(2012), http://www.jstor.org/stable/41486069. CoNLL-U Format, https://universaldependencies.org/format.html.Seventh International Conference on Language Resources
and Evaluation (LREC`10), 2010, https://aclanthology.org/L10-1087/. Conference on Language Technologies and Digital Humanities
(JT-DH-2016), 2016, https://doi.org/10.5281/zenodo.14165095.
It must be noted that not all tasks are available for
every supported language and variety. For instance, semantic role labelling currently
relies on the JOS annotation system for dependency parsing of Slovenian and is thus
only available for annotation of Slovenian datasets but should become available for
Croatian in the future as there are training data available.
An earlier version of this overview was already
presented at the Language Technologies and Digital Humanities conference in
2024,
The Stanza neural pipeline is centred around a
bidirectional long short-term memory (Bi-LSTM) network architecture.
On the level of tokenization and sentence segmentation,
Stanza uses a joint tokenization and sentence segmentation model based on machine
learning. We generally view such learnt tokenizers as suboptimal, since training data
for the two tasks is always limited in size and thus too few tokenization and
sentence-splitting phenomena can be learnt by the model during the training process.
Due to this drawback, CLASSLA-Stanza implements rule-based tokenizers, which handle
both the task of tokenization as well as sentence segmentation. As stated in the
introduction, the two tokenizers used are the Obeliks tokenizer for standard
Slovenian
CLASSLA-Stanza also adds support for the use of external
inflectional lexicons, which is not present in Stanza. For morphologically rich
languages, applying this resource to the annotation process usually significantly
increases the performance of the model.
Most languages support external lexicon use only during lemmatization, except for Slovenian, which supports lexicon use also during morphosyntactic tagging. In that case, the lexicon is put into operation during the tag prediction phase, when the model limits the possible predictions to only those tags that are present in the inflectional lexicon for the specific token. Lexicon usage during lemmatization is similar in both Stanza and CLASSLA-Stanza, the main difference being that Stanza builds a lexicon only from the Universal Dependencies training data, while CLASSLA-Stanza can additionally exploit an inflectional lexicon. Both Stanza and CLASSLA-Stanza use the lexicon for an initial lemma lookup and fall back to predicting the lemma only in case that the form with the corresponding tag is not present in the lexicon. One important difference in the lexicon lookup in CLASSLA-Stanza is that the lookup uses XPOS tags, which contain the full morphosyntactic information, while Stanza uses only the UPOS tag, which is not enough for an accurate lemma lookup in morphologically rich languages.
When training models, Stanza uses a Universal Dependencies dataset as training data for training all the tasks in the pipeline and thus does not enable the user to train models on additional datasets. For certain layers, however, such as lemmatization and morphosyntactic tagging, the South Slavic languages often have more training data than available for dependency parsing, which is exploited by CLASSLA-Stanza. Thus, for example, instead of using only the 210 thousand tokens of data that are used for training the dependency parser, the latest set of standard Croatian models in CLASSLA-Stanza includes morphosyntactic tagging and lemmatization models which were trained on an additional 290 thousand tokens, manually annotated only on these two levels of annotation.
CLASSLA-Stanza also has a special way of handling
“closed-class” tokens. Closed-class control is a feature of the tokenizers and
ensures that punctuation and symbols are assigned appropriate morphosyntactic tags
and lemmas. It also prevents other tokens that are not defined as punctuation and
symbols in the tokenizer from being annotated as such. In addition to punctuation and
symbols, the Slovenian package also includes closed-class control for pronouns,
determiners, prepositions, particles, and coordinating and subordinating
conjunctions. These additional closed classes are controlled during the
morphosyntactic tagging phase using the inflectional lexicon as a reference,
disallowing any token to be labelled with a closed class label if it was not defined
as such in the lexicon.
The Stanza pipeline relies on pretrained word embeddings
as an underlying resource. While it uses embedding collections based on Wikipedia
data, CLASSLA-Stanza goes the extra mile by using the CLARIN.SI embeddings,23rd
Annual Conference of the European Association for Machine Translation
(EAMT), 2022, https://aclanthology.org/2022.eamt-1.41/.
When working with Slovenian, Croatian, or Serbian, the
pipeline can be set to any of the four predetermined settings which are used for
processing different varieties of the same language. These settings are called modes and can be either standard, nonstandard, or web. For Slovenian, an
additional spoken mode is available. The processing modes
determine which model is used on every level of annotation and are associated with
their respective language varieties. The reasons for introducing separate processing
modes for spoken and web texts are described in Sections 5.2 and 5.3. Below is an
overview showing which model is used on every layer for every mode:
The reason why the nonstandard and the web processing modes use the standard dependency parsing model is primarily the lack of training data for training a model beyond standard text and spoken transcripts. The lack of motivation for building a dataset for parsing nonstandard text lies in the fact that the parsing model has upstream lemma and morphosyntactic information at its disposal and therefore requires dedicated training data to a much lesser extent than those upstream processes.
To illustrate the performance of CLASSLA-Stanza, Table 3
provides a comparison of the results produced by both Stanza and CLASSLA-Stanza when
generating predictions using the Slovenian standard models on the SloBENCH evaluation
dataset.1
scores, while the relative error reduction between the scores of the pipelines is
presented in percentages.
Despite CLASSLA-Stanza originating as a fork of Stanza, there are currently no plans to merge CLASSLA-Stanza with the original Stanza project, as CLASSLA-Stanza is intended as a separate project with a different focus. While Stanza takes a broader approach, aiming to achieve good performance across a wide range of different languages, CLASSLA-Stanza focuses more on language-specific solutions that improve performance for the South Slavic languages in particular.
The latest models included in the 2.2 release of CLASSLA-Stanza were trained on a variety of datasets in five different languages: Slovenian, Croatian, Serbian, Macedonian, and Bulgarian. Slovenian had three types of training datasets available—a standard training dataset, a nonstandard training dataset, and a spoken training dataset. Croatian and Serbian were associated with two training datasets—one consisting of standard-language texts and one consisting of nonstandard texts—while Bulgarian and Macedonian only had a standard-language training dataset available.
Slovenian standard language models were first trained
using the 1.0 version of the SUK training corpus.
Nonstandard Slovenian models were trained on a
combination of the standard training corpus and the nonstandard Janes-Tag training
corpus,
Slovenian spoken models were trained on a combination of
the SUK corpus and the Spoken Slovenian UD Treebank,Tenth International
Conference on Language Resources and Evaluation (LREC`16), 2016, https://aclanthology.org/L16-1248/.
Croatian standard language models were trained on the
hr500k training corpus,
Serbian standard models were trained on the Serbian
portion of the SETimes corpus,
Serbian nonstandard models were trained, similar to the
previously introduced languages, on a combination of the standard dataset and the
nonstandard ReLDI-NormTagNER-sr training corpus.
Macedonian standard models were trained on a corpus made
up of the Macedonian version of the MULTEXT-East “1984” corpus1984 by George Orwell in approximately 113 thousand tokens, while the
SETimes.MK corpus in its 0.1 version is made up of 13,310 tokens of news
articles.
Bulgarian standard models were trained on the
BulTreeBank training corpus,5th Workshop on Balto-Slavic
Natural Language Processing, 2015, https://aclanthology.org/W15-5313/.
Table 4 provides an overview of dataset sizes for every language, variety, and annotation layer.
In this section, the model training process is described
in detail. Only a descriptive account of the process is provided here. For a list of
the specific commands and oversampling scripts used, refer to the GitHub repository
of the training procedure.GitHub - clarinsi/classla-training: Training scripts for the
CLASSLA pipeline,
https://github.com/clarinsi/classla-training.
In this paper, we give the general overview of the
process which is common to all supported languages. For the specific steps that are
unique to each language, please refer to the CLASSLA-Stanza technical report, a
longer and older version of this paper available on arXiv.arXiv
preprint (2023), https://doi.org/10.48550/arXiv.2308.04255.
The illustration of the basic procedure that was used to train standard models for the levels of morphosyntactic tagging, lemmatization, and dependency parsing for the latest release of CLASSLA-Stanza is shown in Figure 1.
As stated in the introduction, all tokenizers used by CLASSLA-Stanza are rule-based and thus do not need to be trained. Model training is thus performed on pretokenized data, typically beginning on the level of morphosyntactic tagging and continuing on through the subsequent annotation layers.
To ensure realistic evaluation results, automatically generated upstream annotations, rather than manually assigned annotations, were used as validation and test dataset inputs on each layer. For this, empty validation and test datasets first had to be generated by stripping all annotations from the test and validation datasets on all levels except for tokenization. These empty files were filled with model-generated annotations on each level, so that validation and model evaluation on subsequent layers could be performed on automatically generated upstream labels. Training datasets were not annotated with automatically generated upstream labels, since it is unclear whether this would lead to any performance gains and would require a more complicated type of cross-validation method such as jackknifing (splitting the data into N bins, training a model on N-1 bins, and annotating the N-th bin, repeating the process N times).
For each language, standard models were first trained. For morphosyntactic tagging, the training and validation datasets from the prepared three-way data split along with the pretrained word embeddings were used as inputs to the tagger module. After training, the tagger was used in predict mode to generate predictions on the empty test dataset and evaluate the performance of the tagger. After predictions were made for the test set, predictions were generated in the empty validation dataset as well to produce a validation file with automatically generated morphosyntactic labels, that can be used later during training of subsequent annotation layer models, such as those for lemmatization and dependency parsing.
Once morphosyntactic predictions and evaluation results were obtained, the lemmatizer was trained. The validation and training datasets were used as inputs. In addition, for most languages, the inflectional lexicon is also provided to the lemmatizer as an underlying resource. During training, the lexicon is stored in the lemmatization model file to act as an additional controlling element during lemmatization. After training, the lemmatizer was run in predict mode to obtain evaluation results and add lemma predictions to the validation and test datasets for the training of the dependency parser model.
The dependency parser model was trained after lemmatizer
training was finalized. CLASSLA-Stanza currently supports two types of annotation
systems for syntactic dependency annotation: the UD dependency parsing annotation
system, which is available for all supported languages except Macedonian, and the JOS
parsing system, which is only available for Slovenian.
The process for training models for named-entity recognition was quite similar to the other tasks. The tagger training here accepts pretrained word embeddings and training and validation datasets as underlying resources. After training, the named entity recognition tagger can be run in predict mode to obtain evaluation results.
The spoken models were trained using the same process as the standard models, and a similar process was also followed for the nonstandard models with a few notable exceptions. Firstly, no syntactic dependency annotations are present in the nonstandard datasets. As a result, no nonstandard dependency parsing models were trained. Before training the nonstandard models, approximately 20% of diacritics were removed from the training datasets to ensure that the models will learn to effectively handle dediacritized forms, which occur prominently in online communication.
As noted in Section 2, CLASSLA-Stanza significantly outperforms Stanza on the Slovenian benchmark, with the relative error reduction between 34% and 98%, depending on the processing layer.
However, in order to fully assess the performance of the newly-trained models, we conduct a series of additional performance analyses in this section. In Section 5.1, we give a detailed rundown of the performance of the models for various UPOS and syntactic dependency labels for each language. In Section 5.2, we present experiments using various Slovenian models to annotate spoken data and discuss why the training of models specifically dedicated to annotating spoken transcripts was justified. Finally, in Section 5.3 we present a more qualitative investigation into the performance of the models on web-specific data.
To obtain a sense of which categories a model
struggles with and which ones it handles well, model predictions for specific UPOS
and UD syntactic relations were inspected. An accuracy score was calculated for
all 17 UPOS labels and the 12 most frequent UD syntactic relations in the Croatian
hr500k training corpus.
The highest accuracies among UPOS tags are generally
found with tags that represent function word classes, such as AUX (auxiliaries), ADP (adpositions), and PRON (pronouns), and closed-class tags, such as PUNCT (punctuation) and SYM
(symbols), which are handled by the pipeline, inter alia, through rules in the
tokenizer, as described in Section 2. Conversely, the lowest accuracies are found
with the infrequent INTJ tag (interjections)—of which there
were only 5 instances in total in the Slovenian standard test dataset and no
instances at all in the Serbian standard test dataset—and the loosely delineated
X tag, which is used for certain abbreviations,
A similar trend is found among the UD syntactic
relations. Relations such as case (which usually connects
nominal heads with adpositions), cc (connects conjunct
heads with coordinating conjunctions), and aux (connects
verbal heads with auxiliary verbs) are used for fixed grammatical patterns that
permit little variation. These display consistently high accuracies across all
languages. Somewhat lower accuracies are displayed by the obl relation, mostly used for oblique nominal arguments which play a less
central role in the sentence structure than the core verbal arguments. It has been
found that previous versions of dependency parsing models for CLASSLA-Stanza often
incorrectly assigned the obj relation (used for direct
objects) to instances which should receive the obl relation
and vice versa.Slovenščina
2.0: empirične, aplikativne in interdisciplinarne raziskave 11, No. 1
(2023): 218–46, https://doi.org/10.4312/slo2.0.2023.1.218-246.obl and obj relations in other languages as well.
The Slovenian spoken parsing model noticeably stands
out, as there is a clear drop with the nmod, nsubj, obl, conj,
root, and to some degree also the obj relations. A
subsequent inspection of the model’s predictions in these cases revealed that
these errors can be ascribed to the highly fragmentary nature of spoken language,
causing the model to produce errors when trying to annotate fragments for which
the exact role in the wider sentence structure is more difficult to determine
unambiguously. This prompted the question of how the performance of the Slovenian
spoken models compares to that of the standard and nonstandard models when tasked
with annotating transcripts of spoken language. We explore this question in the
following section.
Even though transcripts of spoken Slovenian, manually
annotated on the levels of morphosyntactic tagging, lemmatization, and dependency
parsing, have been available for quite some time already in the form of the Spoken
Slovenian UD Treebank,Conference on Language
Technologies and Digital Humanities (JT-DH-2024), 2024, https://doi.org/10.5281/zenodo.13936394. Jaka Čibej and Tina Munda,
“Metoda polavtomatskega popravljanja lem in oblikoskladenjskih oznak na primeru
učnega korpusa govorjene slovenščine ROG” [A Method for Semi-automatic
Corrections of Lemmas and Morphosyntactic Tags: The Case of the ROG Training
Corpus of Spoken Slovene], paper presented at the Conference
on Language Technologies and Digital Humanities (JT-DH-2024), 2024,
https://doi.org/10.5281/zenodo.13936390.
However, it remained to be tested whether the newly
trained models truly offer any considerable advantages to the already available
standard and nonstandard models when used to annotate transcripts of spoken
Slovenian. To investigate this, all three models currently available for Slovenian
were evaluated on the test set of the Spoken Slovenian Treebank. The models were
evaluated on the levels of morphosyntactic tagging, lemmatization, and UD
dependency parsing. The results of this comparison experiment are shown in Table
7.1
score for all three types of morphosyntactic labels combined (UPOS, XPOS, and
UFeats), for the lemmatization model using the micro F1 score for all lemmas, and for the dependency parsing model using the
micro F1 of the commonly employed labelled attachment
score, or LAS score. The LAS score gives the percentage of tokens with both a
correctly assigned head token and a correctly assigned dependency
label.
The results show that the spoken models clearly
outperform the other two sets of models in all three tasks. This increase in
performance is particularly evident with the dependency parsing models, suggesting
that the differences between spoken and written language in Slovenian are most
pronounced at the syntactic level, which is a finding also reported by Dobrovoljc
and Čibej.UniDive
3rd General
Meeting: Universality, diversity and idiosyncrasy in language
technology, 2025,https://unidive.lisn.upsaclay.fr/lib/exe/fetch.php?media=meetings:general_meetings:3rd_unidive_general_meeting:59_spoken_slovenian_treebank_n.pdf.
To facilitate the use of spoken models, a special spoken processing mode was added to the Slovenian pipeline
in version 2.2 that combines the standard tokenizer and spoken models for all
subsequent layers of annotation.
The model evaluations described in Section 5.1
provide a good summary of how well the CLASSLA-Stanza pipeline performs on both
purely standard and purely nonstandard data. However, modern corpus construction
techniques—especially for low-resource languages—often rely on crawling data from
online conversations, articles, blogs, etc.,
The CLASSLA-Stanza tool was used with the
newly-trained models to add linguistic annotations to the CLASSLA-web corpora,
which consist of texts crawled from the internet domains of the corresponding
languages.
Quite a few of the analysed differences in the model
outputs were connected to the processes of sentence segmentation and tokenization.
In the CLASSLA-Stanza annotation pipeline, both processes are controlled by the
tokenizer. As stated in Section 2, the pipeline uses two different tokenizers
depending on the language and the processing mode used.„Svaku našu riječ
treba da čuvamo kao najveće blago.“ was split into two segments—the first
ending on the period character, while the quotation mark was moved to a separate
sentence:
# newpar id = 76
# sent_id = 76.1
# text =
„ Svaku našu riječ treba da čuvamo kao
najveće blago.
1 „
2
Svaku
3 našu
4 riječ
5 treba
6 da
7 čuvamo
8
kao
9 najveće
10 blago
11 .
# sent_id = 76.2
# text
= “
1 “
The nonstandard models handled nonstandard word forms
quite a bit better than the standard models. Particularly problematic for the
standard Slovenian models were forms with missing diacritics, such as “sel”
instead of šel, “cist” instead of čisto, “hoce” instead of hoče, and “clovek”
instead of človek. These were often assigned incorrect
lemmas and morphosyntactic tags. An example of the standard lemmatiser output for
the word form “hoce” (which corresponds to hoče in standard
Slovenian (Eng. “he/she/it wants”)) is displayed below. The model invents a
nonexistent lemma “hocati”, while the correct form should be the standard
Slovenian hoteti:
# sent_id = 53.1
# text = lev je lev pa naj
govori kar kdo hoce
1 lev lev
2 je biti
3 lev lev
4 pa pa
5 naj naj
6 govori govoriti
7 kar kar
8 kdo kdo
9 hoce
hocati
Nonstandard forms which do not differ much from their standard counterparts, such as “zdej” as opposed to “zdaj” and “morš” as opposed to “moraš”, were generally handled well by both sets of models and did not cause many discrepancies in the outputs.
The analysis of such differences in the model outputs
showed that the best results for the web corpus were achieved on the one hand by
the standard tokenizer, and on the other by the nonstandard models for all
subsequent levels of annotation. In light of this, a new web mode was implemented for the CLASSLA-Stanza pipeline. This new mode
combines the standard tokenizer and nonstandard models for the other layers in a
single package and is intended specifically for the annotation of texts
originating on the Internet.
In this paper, we provided an overview of the
CLASSLA-Stanza pipeline for linguistic processing of the South Slavic languages and
described the training process for the models included in the latest release of the
pipeline. We described the main design differences to the Stanza neural pipeline,
from which CLASSLA-Stanza arose as a forked project. We provided a summary of the
model training process, while the technical documentation
CLASSLA-Stanza gives consistent results across all
supported languages and outperforms the Stanza pipeline on all supported NLP tasks,
as illustrated in Sections 2 and 4. However, low accuracies are still seen for
infrequent labels and pairs of labels that are not easily disambiguated. It remains
to be seen whether larger and more diverse training datasets can contribute to
improving model performance in these specific cases, or rather the move to contextual
embeddings, i.e., transformer models. The newly included spoken models perform much
better on transcripts of spoken language, and they can be easily deployed using the
special spoken processing mode implemented within
CLASSLA-Stanza. Additionally, when processing texts obtained from the Internet,
special care must be taken to use the combination of models that is best suited for
the task, which is why we also described the special web
processing mode.
The release of a specialized pipeline for linguistic
processing of South Slavic languages is an important new milestone in the development
of digital resources and tools for this relatively under-resourced group of
languages. However, there is still much left to be achieved and improved upon. Full
support for all annotation tasks and modalities, such as, for instance, semantic role
labelling and the spoken modality, remains to be extended to other languages as well.
As larger training datasets become available, more capable models can be trained for
the currently supported languages. In addition, the aim is also to extend support to
other members of the South Slavic language group, provided that training datasets of
sufficient size are eventually produced for those languages as well. Finally, the
performance of the CLASSLA-Stanza pipeline should also be compared to other recent
state-of-the-art tools for automatic linguistic annotation, such as Trankit,
The work described by this paper was made possible by
the Development of Slovene in a Digital Environment project (Razvoj slovenščine v
digitalnem okolju, project ID: C3340-20-278001), financed by the Ministry of Culture
of the Republic of Slovenia and the European Regional Development Fund, the Language
Resources and Technologies for Slovene research program (project ID: P6-0411), the
MEZZANINE project (Basic Research for the Development of Spoken Language Resources
and Speech Technologies for the Slovenian Language, project ID: J7-4642), the SPOT
project (A Treebank-Driven Approach to the Study of Spoken Slovenian, Z6-4617), and
the Large Language Models for Digital Humanities project (Grant GC-0002), all
financed by the Slovenian Research Agency, and the CLARIN.SI research
infrastructure.
rdAnnual Conference of the European Association for Machine Translation, EAMT, 2022. https://aclanthology.org/2022.eamt-1.41/.
Conference on Language Technologies and Digital Humanities (JT-DH-2024),2024. https://doi.org/10.5281/zenodo.13936390.
Computational Linguistics47, No. 2 (07 2021): 255–308. https://doi.org/10.1162/coli_a_00402.
Conference on Language Technologies and Digital Humanities (JT-DH-2024),2024. https://doi.org/10.5281/zenodo.13936394.
UniDive 3
rd
General Meeting: Universality, diversity and idiosyncrasy in language technology, 2025. https://unidive.lisn.upsaclay.fr/lib/exe/fetch.php?media=meetings:general_meetings:3rd_unidive_general_meeting:59_spoken_slovenian_treebank_n.pdf.
Tenth International Conference on Language Resources and Evaluation (LREC`16),2016. https://aclanthology.org/L16-1248/.
Third Workshop on Universal Dependencies (UDW, SyntaxFest 2019),2019. https://doi.org/10.18653/v1/W19-8004.
Slovenščina 2.0: empirične, aplikativne in interdisciplinarne raziskave11, No. 1 (2023): 218–46. https://doi.org/10.4312/slo2.0.2023.1.218-246.
Seventh International Conference on Language Resources and Evaluation (LREC`10),2010. https://aclanthology.org/L10-1087/.
Language Resources and Evaluation46, No. 1 (2012): 131–42. https://doi.org/10.1007/s10579-011-9174-8.
Collaboration and Computing for Under-Resourced Languages (CCURL)workshop, 2016. http://www.lrec-conf.org/proceedings/lrec2016/workshops/LREC2016Workshop-CCURL2016_Proceedings.pdf#page=74.
Eighth Language Technologies Conference, 2012. https://doi.org/10.5281/zenodo.14165686.
thWorkshop on Balto-Slavic Natural Language Processing, 2019. https://doi.org/10.18653/v1/W19-3704.
CLARIN. The Infrastructure for Language Resources, edited by Darja Fišer and Andreas Witt. De Gruyter, 2022. https://doi.org/10.1515/9783110767377-017.
Conference on Language Technologies and Digital Humanities (JT-DH-2024),2024. https://doi.org/10.5281/zenodo.13936406.
5
th
Workshop on Balto-Slavic Natural Language Processing, 2015. https://aclanthology.org/W15-5313/.
58
th
Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020. https://doi.org/10.18653/v1/2020.acl-demos.14.
5
th
Workshop on Balto-Slavic Natural Language Processing, 2015. https://aclanthology.org/W15-5306/.
arXiv preprint(2023). https://doi.org/10.48550/arXiv.2308.04255.
arXiv preprint(2021). https://doi.org/10.48550/arXiv.2101.03289.
Predstavljamo CLASSLA-Stanza, orodje za učinkovito jezikovno obdelavo besedil v naravnem jeziku, ki podpira več južnoslovanskih jezikov in je posebej prilagojeno zanje. Najnovejša različica orodja podpira obdelavo besedil v slovenščini, hrvaščini, srbščini, bolgarščini in makedonščini. Orodje je nastalo kot razvejitev cevovoda Stanza za jezikovno obdelavo, z vrsto novih izboljšav. V tem članku opisujemo glavne razlike med CLASSLA-Stanza in Stanza, podajamo pregled procesa usposabljanja za izdelavo modelov, vključenih v najnovejšo različico orodja 2.2, podajamo pregled zmogljivosti najnovejših modelov in razpravljamo o učinkovitosti cevovoda za označevanje govorjenih besedil in besedil, ki izvirajo z interneta.
CLASSLA-Stanza ohranja večino arhitekture in zasnove Stanza, vendar uvaja nekaj ključnih sprememb, med drugim specializiran niz pravilnih tokenizatorjev, ki obdelujejo segmentacijo in tokenizacijo stavkov, podporo za uporabo zunanjih fleksijskih leksikonov kot dodatnega kontrolnega elementa med napovedovanjem, specializiran način obdelave besed zaprtega razreda in podporo za več dodatnih jezikovnih različic, kot so nestandardni jezik, govorjeni jezik in besedila, pridobljena z interneta.
Splošni proces usposabljanja modela za CLASSLA-Stanza sledi zaporednemu postopku, pri katerem se usposabljanje izvaja na predhodno tokeniziranih podatkih, model pa se usposablja za vsako plast označevanja. Po usposabljanju modela za vsako plast se ta model uporabi za samodejno generiranje navzgornjih označb za validacijske in testne podatkovne nize, ki se nato uporabijo med usposabljanjem in ocenjevanjem modelov na naslednjih plasteh označevanja. Med zadnjim krogom usposabljanja modelov so bili modeli najprej usposobljeni za morfosintaktično označevanje, nato za lematizacijo in nazadnje za razčlenjevanje odvisnosti. Med usposabljanjem modela za lematizacijo je bil modulu za usposabljanje na voljo fleksijski leksikon, medtem ko slovenščina podpira tudi uporabo fleksijskega leksikona med usposabljanjem morfosintaktičnega označevalca. Usposabljanje nestandardnih in govornih modelov je potekalo po podobnem postopku z nekaj manjšimi odstopanji.
Predstavljamo ocene natančnosti modelov za vse jezike na različnih oznakah v naborih oznak UD Part-of-Speech in UD dependency relation. Pri oznakah Part-of-Speech so najvišje natančnosti na splošno ugotovljene pri razredih funkcionalnih besed, najnižje pa pri redki oznaki INTJ in oznaki X, ki je slabo opredeljena. Pri odvisnostnih odnosih odnosi case, cc in aux kažejo dosledno visoko natančnost v vseh jezikih, medtem ko nekoliko nižjo natančnost kaže odnos obl, ki se večinoma uporablja za posredne nominalne argumente, ki imajo v stavčni strukturi manj osrednjo vlogo kot glavni verbalni argumenti.
Prav tako ponujamo analizo na novo usposobljenih govornih modelov, ki so na voljo za slovenski jezik v najnovejši različici CLASSLA-Stanza. Primerjali smo slovenske govorne morfosintaktične modele označevanja, lematizacije in odvisnostnega razčlenjevanja z zmogljivostjo ustreznih standardnih in nestandardnih modelov pri označevanju transkriptov govorjenega jezika. Rezultati kažejo, da govorni modeli znatno presegajo standardne in nestandardne modele. Usposabljanje modelov, ki so posebej prilagojeni govorjenemu jeziku, je bilo zato upravičeno in bi ga bilo treba v prihodnosti razširiti na druge jezike.
Opravili smo tudi analizo, da bi ocenili, kako dobro modeli CLASSLA-Stanza obdelujejo besedila, ki izvirajo z interneta. Ugotovili smo, da so standardni tokenizatorji bolje obdelovali segmentacijo stavkov, nestandardni modeli pa nestandardne oblike, ki se pojavljajo v internetnem jeziku, na vseh drugih ravneh označevanja. Posledično je bil implementiran nov način obdelave spleta, ki je vključen v najnovejšo različico procesa.