Classes and Methods

AuDoLab.AuDoLab main module

class AuDoLab.AuDoLab.AuDoLab

Bases: object

abstract_scraper(type='arxiv', url=None, pages=2, prepro=False, ngram_type=2)

Scrapes the pages arxiv.org, pubmed.gov for paper abstracts

Args:

type (str, optional): “arxiv” or “pubmed”. Defines for which page

the scraping is done. Defaults to “arxiv”.

url (str, optional): The given url after which the papers are

scraped. Must be in line with type. Defaults to None.

pages (int, optional): Number of pages that are iterated over.

Defaults to 2.

prepro (bool, optional): If True, the scraped documents are

preprocessed directly. Defaults to False.

ngram_type (int, optional): Number of ngrams in preprocessing.

Defaults to 2.

Returns:

pd.DataFrame: DataFrame with the stored abstracts

choose_classifier(df, classifier, i)

Returns dataframe where documents that are classified to target class have 1, otherwise, 0.

Args:

df (pd.Dataframe): dataframe of target documents

classifier (list): list of all possible o-svm classifiers

i (int): index of which classifier is chosen/preferred

Returns:

pd.dataframe: documents that are classified as belonging to target

get_ieee(url=None, keywords=None, operator='OR', pages=2, in_data='author', prepro=False, ngram_type=2)

Function to scrape abstracts of scientific papers from the givin url.

We used https://ieeexplore.ieee.org/search/advanced to generate a list like https://ieeexplore.ieee.org/search/searchresult.jsp?action=se arch&newsearch=true&matchBoolean=true&queryText=(%22Author%20Keywords%2 2:cotton)&highlight=true&returnFacets=ALL&returnType=SEARCH&matchPubs=T rue&rowsPerPage=100&pageNumber=1 with the search results. The abstracts of the papers listet on that list of search results will be stored in a .txt file with the givin file name.

Args:

url (str, optional): The url of the website, whos presented paper

abstracs will be scraped. Defaults to None.

keywords (list, optional): List of keywords that are searched

for. Defaults to None.

keywords (iist, optional): keywords that are searched for.

Defaults to None.

operator (str, optional): Operator between the keywords.

“AND” or “OR”. If “AND” the search results must include all keywords. Defaults to “OR”.

pages (int, optional): Number of pages that are iterated over.

Translates directly to number of abstracts that are scraped. Roughly there are 100 abstracts scraped per page. Defaults to 2.

in_data (str, optional): If the keywords are searched for in the

author keywords or in all metadata. Defaults to “author”.

prepro (bool, optional): if True, the scraped data will directly be

preprocessed for later use. Defaults to False.

ngram_type (int, optional): number of ngrams in preprocessing.

Defaults to 2.

Returns:

pd.DataFrame: DataFrame with the stored abstracts and metadata

lda_modeling(data, num_topics, corpus=None, dict=None, no_below=None, no_above=None, random_state=101, passes=20, chunksize=500, eta='auto', eval_every=None, multi=True, alpha='asymmetric', column='preprocessed')

The function performs lda modelling as described in this https://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf paper.

Args:

corpus (iterable of list of (int, float), optional): Stream of

document vectors or sparse matrix of shape

num_topics (int): pre-defined number of topics

id2word ({dict of (int, str):

gensim.corpora.dictionary.Dictionary}): Mapping from word IDs to words. It is used to determine the vocabulary size, as well as for debugging and topic printing.

random_state (int): for recreating exact identical output. Defaults

to 101.

passes (int): Number of passes through the corpus during training.

Defaults to 20.

chunksize (int, optional): chunksize in lda passes. Defaults to

eta (str, optional): [description]. Defaults to “auto”.

eval_every ([type], optional): Hyperparameter in LDA used to

initiliaze the Dirichlet distribution. Defaults to None.

multi (bool, optional): If true, the in gensim incorporated

multicore variant is used. Defaults to True.

alpha (str, optional): OTher Dirichlet Prior. Defaults to

“asymmetric”.

Returns:

lda_model: returns lda_model output

lda_visualize_topics(save_name='audolab_model.png', lda_model=None, bow_corpus=None, dictionary=None, type='pyldavis', figsize=(20, 10), facecolor='k', width=1600, height=800, background_color='white', topic=0, words=100, save=False, n_clouds=1)

Visualizes the topic models output in wordclouds or pyldavis

Args:

lda_model (gensim.models.ldamodel.LdaModel): the created LDA model

bow_corpus (gensim.corpora.dictionary.Dictionary): Bag of words

corpus of used documents

dictionary (gensim.corpora.dictionary.Dictionary): Dictionary of

all words

save_name (str, optional): name under which the plots should be

save. Defaults to “audolab_model.png”.

type (str, optional): type of visualisation- either “clouds” or

“pyldavis”. Defaults to “clouds”.

figsize (tuple, optional): Size of wordclouds. Defaults to

(50, 30).

facecolor (str, optional): Colour of wordcloud Defaults to “k”.

width (int, optional): width of plots. Defaults to 2000.

height (int, optional): height of plots. Defaults to 1000.

background_color (str, optional): Background colour of wordcloud.

Defaults to “white”.

topic (int, optional): IF only one wordcloud is plotted, index of

topic that is plotted. Defaults to 0.

words (int, optional): Number of words per cloud. Defaults to 100.

save (bool, optional): whether the plots should be saved or not.

Defaults to False.

n_clouds (int, optional): Number of word clouds that are plotted.

Defaults to 1.

Raises:

ValueError: If save_name is not a string: no “Please specify a

string as the name under which the plots should be saved”

one_class_svm(training, predicting, nus, quality_train=0.85, min_pred=0.05, max_pred=0.2, gamma='auto', kernel='rbf')

Returns the classifiers that fullfill the required conditions.

Args:

training (DataFrame): training dataset of preprocessed documents

predicting (DataFrame): target dataset of preprccessed documents

nus (list of floats): hyperparameters over which are looped. For

each nu the classifier is trained

quality_train (float, optional): percentage of training data that

seems to belong to target class. Default: 0.85. Defaults to 0.85.

min_pred (float, optional): percentage of target data that has to

be at least classified as belonging to target class for classifier to be considered. Default: 0.0. Defaults to 0.05.

max_pred (float, optional): percentage of target class that is

maximally allowed to be classified as belonging to

target class for classifier to be considered.. Defaults to 0.2.

gamma (str, optional): Hyperparamter of O-SVM. Defaults to “auto”.

kernel (str, optional): Kernel function used in O_SVM. Defaults to

“rbf”.

Returns:

pd.DataFrame: DataFrame with stored classifiers that fulfill

conditions

text_cleaning(data, column, ngram_type=2)

The data will be lemmatized, tokenized and the stopwords will be deleted.

Args:

data (pd.DataFrame): Dataframe where the documents to be

preprocessed are stored

column (str): Column name of the column where docs are stored

ngram_type (int, optional): Number of ngrams used. Defaults to 2.

Returns:

pd.DataFrame: DataFrame where the original docus and the

preprocessed documents are stored

tf_idf(data, papers, data_column, papers_column, features=None, ngrams=2)

Creates tf-idf objects for one-class SVM classification.

The tf-idf scores are calculated over a joint corpus, however the target data and the out-of-domain training data are stored in seperate, as the one-class SVM is only trained on the tf-idf scores of the out-of-domain training data.

Args:

data (DataFrame): preprocessed target documents

papers (DataFrame): preprocessed out-of-domain training data

data_colum (String): name of columnin target dataframe where

lemmatized documents are stored. Defaults to ‘lemma’

papers_colum (String): name of column in out-of-domain training

dataframe where lemmatized documents are stored. Defaults to ‘lemma’

ngrams (int, optional): whether ngram are formed.

Defaults to 2.

features (int, optional): number of max features.

Defaults to 8000.

Returns:

data and papers: tfidf object data for target data and

out-of-domain training data

AuDoLab.AuDoLab.warn(*args, **kwargs)

Subpackages

• AuDoLab.subclasses.abstractscraper
• AuDoLab.subclasses.abstractscraper_arxiv
• AuDoLab.subclasses.abstractscraper_pubmed
• AuDoLab.subclasses.preprocessing
• AuDoLab.subclasses.tf_idf
• AuDoLab.subclasses.one_class_svm
• AuDoLab.subclasses.lda