AuDoLab.subclasses.abstractscraper

class AuDoLab.subclasses.abstractscraper.AbstractScraper[source]

Bases: object

Abstractscraper - scrapes through IEEE Xplore and, depending on the given search query, scrapes the abstracts of scientific papers

async get(url=None, keywords=None, operator='OR', pages=2, in_data='author')[source]
Simply runs the self._open function that open ieeeXplore and seraches

for paper urls.

Args:
url (str, optional): when the user specifies an own search query on

IEEEXplore. Defaults to None.

keywords (iist, optional): keywords that are searched for. Defaults

to None.

operator (str, optional): [description]. Defaults to “OR”.

pages (int, optional): Number of pages the algorithm iterates over.

Defaults to 2.

in_data (str, optional): “author” or “all_meta” whether to search in

author keywords or all metadata. Defaults to “author”.

async get_abstracts(url=None, keywords=None, operator='OR', pages=2, in_data='author', features=['abstract', 'title', 'citationCount', 'doi', 'totalDownloads', 'keywords', 'publicationYear', 'authors'])[source]

Function to scrape abstracts of scientific papers from the givin url.

We used https://ieeexplore.ieee.org/search/advanced to generate a list like https://ieeexplore.ieee.org/search/searchresult.jsp?action=se arch&newsearch=true&matchBoolean=true&queryText=(%22Author%20Keywords%22 :cotton)&highlight=true&returnFacets=ALL&returnType=SEARCH&matchPubs=Tru e&rowsPerPage=100&pageNumber=1 with the search results. The abstracts of the papers listet on that list of search results will be stored in a .txt file with the givin file name.

Args:
url (str, optional): when the user specifies an own search query on

IEEEXplore. Defaults to None.

keywords (iist, optional): keywords that are searched for. Defaults

to None.

operator (str, optional): [description]. Defaults to “OR”.

pages (int, optional): Number of pages the algorithm iterates over.

Defaults to 2.

in_data (str, optional): “author” or “all_meta” whether to search in

author keywords or all metadata. Defaults to “author”.

features (list, optional): which features should be scraped.

Defaults to [“abstract”, “title”, “citationCount”, “doi”, “totalDownloads”, “keywords”, “publicationYear”, “authors”].

Returns:
pd.DataFrame: DataFrame containing the paper abstracts and the

selected features

AuDoLab.subclasses.abstractscraper.warn(*args, **kwargs)[source]