Importer Module for CRAN

This module imports R packages published at the Comprehensive R Archive Network (CRAN).

Specifically, it reads the table of packages ordered by date of publication. This table contains for each R package the package name, title and date of publication. Based on the package name, each package url can be accessed from:

https://cran.r-project.org/web/packages/<package_name>/index.html

Several attributes are listed for each package. Among them, the following attributes are imported, when present, to the MaRDI knowledge graph:

Version:

Version of the package.

Depends:

Software and package dependencies, including other R packages.

Published:

Date of publication.

Author:

Authors of the package are to be indicated according to the CRAN Repository Policy with the abbreviation [aut]. Given that this guideline is not always implemented, it is not always possible to properly parse the authors.

When no abbrevations describing the role of each individual are included, just the first listed author is imported.

License:

List of accepted licenses.

Maintainer:

Software maintainer (Generally one of the authors).

mardi_importer.cran.CRANSource class

class mardi_importer.cran.CRANSource.CRANSource(*args, **kwargs)

Bases: ADataSource

Processes data from the Comprehensive R Archive Network.

Metadata for each R package is scrapped from the CRAN Repository. Each Wikibase item corresponding to each R package is subsequently updated or created, in case of a new package.

packages

Dataframe with package name, title and date of publication for each package in CRAN.

Type:

Pandas dataframe

setup()

Create all necessary properties and entities for CRAN

pull()

Reads date, package name and title from the CRAN Repository URL.

The result is saved as a pandas dataframe in the attribute packages.

Returns:

Attribute packages

Return type:

Pandas dataframe

load_people(packages_rds: str | None = None, sparql=None) → None

Read the people of every package from CRAN’s package database, and their items.

Mentions are joined into people across all packages and matched to the person items the packages link now (see mardi_importer.cran.resolve). If the database or the SPARQL store cannot be read, packages fall back to their page and to resolving people without that overview.

new_package(date: str, label: str, title: str) → RPackage

push(since: str | None = None)

Updates the MaRDI Wikibase entities corresponding to R packages.

For each package name in the attribute packages checks if the date in CRAN coincides with the date in the MaRDI knowledge graph. If not, the package is updated. If the package is not found in the MaRDI knowledge graph, the corresponding item is created.

It creates a mardi_importer.cran.RPackage instance for each package.

Parameters:

since – Only packages published on CRAN on or after this day (YYYY-MM-DD).

sync_archive_status(dry_run: bool = False) → dict

Set or remove the end time of every CRAN project statement.

See mardi_importer.cran.archive.

sync_task_views(dry_run: bool = False) → dict

Task view items, and part of on their packages. See mardi_importer.cran.ctv.

mardi_importer.cran.RPackage class

mardi_importer.cran.RPackage.wikidata_r_packages(session=None) → Dict[str, str]

CRAN package name → Wikidata QID, in one query per run.

Items with the package’s CRAN project (P5565) first, then items that are an R package (Q73539779) with the package’s name as English label. A name shared by several items is left out. Empty if Wikidata cannot be reached: the Wikidata QID is then not added, nothing else changes.

class mardi_importer.cran.RPackage.RPackage(date: str, label: str, description: str, long_description: str = '', url: str = '', version: str = '', versions: Tuple[str, str]]=<factory>, entries: List[Entry] | None = None, people: People | None = None, wikidata_ids: Dict[str, str] | None=None, license_data: Tuple[str, str]]=<factory>, dependencies: Tuple[str, str]]=<factory>, imports: Tuple[str, str]]=<factory>, crossref_publications: List[CrossrefPublication] = <factory>, arxiv_publications: List[ArxivPublication] = <factory>, zenodo_resources: List[ZenodoResource] = <factory>, _QID: str = '', _item: MardiItem = None, api: MardiClient | None = None, wdi: WikidataImporter | None = None, crossref: CrossrefSource | None = None, arxiv: ArxivSource | None = None, zenodo: ZenodoSource | None = None)

Bases: object

Class to manage R package items in the local Wikibase instance.

date

Date of publication

Type:

str

label

Package name

Type:

str

description

Title of the R package

Type:

str

long_description

Detailed description of the R package

Type:

str

url

URL to the CRAN repository

Type:

str

version

Version of the R package

Type:

str

versions

Previous published versions

Type:

List[Tuple[str, str]]

entries

People named by the package (authors, maintainer), from CRAN’s package database or, failing that, the package page

Type:

List[mardi_importer.cran.authors.Entry] | None

people

Resolution of people to items, shared by the packages of one run

Type:

mardi_importer.cran.resolve.People | None

license

Software license

dependency

Dependencies to R and other packages

imports

Imported R packages

Type:

List[Tuple[str, str]]

_QID

Package QID

Type:

str

date: str

label: str

description: str

long_description: str = ''

url: str = ''

version: str = ''

versions: List[Tuple[str, str]]

entries: List[Entry] | None = None

people: People | None = None

wikidata_ids: Dict[str, str] | None = None

license_data: List[Tuple[str, str]]

dependencies: List[Tuple[str, str]]

imports: List[Tuple[str, str]]

crossref_publications: List[CrossrefPublication]

arxiv_publications: List[ArxivPublication]

zenodo_resources: List[ZenodoResource]

api: MardiClient | None = None

wdi: WikidataImporter | None = None

crossref: CrossrefSource | None = None

arxiv: ArxivSource | None = None

zenodo: ZenodoSource | None = None

property QID: str

Return the QID of the R package in the knowledge graph.

Searches for an item with the package label in the Wikibase SQL tables and returns the QID if a matching result is found.

Returns:

The entity QID representing the R package.

Return type:

str

property item: MardiItem

Return the Item representing the R package.

Adds also the label and description of the package.

Returns:

MardiClient item

Return type:

MardiItem

exists() → str

Checks if an item corresponding to the R package already exists.

Returns:

Entity ID

Return type:

str

is_updated() → bool

Checks if the Item corresponding to the R package is up to date.

Compares the last update property in the local knowledge graph with the publication date imported from CRAN.

Returns:

True if both dates coincide, False otherwise.

Return type:

bool

pull()

Imports metadata from CRAN corresponding to the R package.

Imports Version, Dependencies, Imports and License and saves them as instance attributes. Authors and maintainer come from entries; only when none were given are they read from the package page.

create() → None

Create a package in the Wikibase instance.

This function pulls the package, inserts its claims, and writes it to the Wikibase instance.

Returns:

None

write() → Dict[str, str] | None

Write the package item to the Wikibase instance.

If the item has claims, it will be written to the Wikibase instance. If the item is successfully written, a dictionary with the QID of the item will be returned.

Returns:

A dictionary with the QID of the written item if successful, or None otherwise.

Return type:

Optional[Dict[str, str]]

insert_claims()

update()

Updates existing WB item with the imported metadata from CRAN.

The metadata corresponding to the package is first pulled from CRAN and saved as instance attributes through pull(). The statements that do not coincide with the locally saved information are updated or subsituted with the updated information.

Uses mardi_importer.wikibase.WBItem to update the item corresponding to the R package.

Returns:

ID of the updated R package.

Return type:

str

replace_statements(prop: str, claims: list) → None

Make claims the statements of prop in the item, to be written with it.

Statements equal to one of claims (value and qualifiers) stay as they are; the others are removed, and the new ones added.

process_claims(data, prop_nr, qualifier_nr=None)

parse_publications(description)

Extracts the DOI identification of related publications.

Identifies the DOI of publications that are mentioned using the format doi: or arXiv: in the long description of the R package.

Returns:

List containing the wikibase IDs of mentioned publications.

Return type:

List

get_last_update()

Returns the package last update date saved in the Wikibase instance.

Returns:

Last update date in format DD-MM-YYYY.

Return type:

str

clean_package_list(table_html)

Processes raw imported data from CRAN to enable the creation of items.

• Package dependencies are splitted at the comma position.

• License information is processed using the parse_license() method.

Parameters:

table_html – HTML code obtained with BeautifulSoup corresponding to the table containing the metadata of the R package imported from CRAN.

Returns:

Dataframe with processed data from a single R package including columns: Version, License, Depends and Imports (processed), Author and Maintainer (as on the page).

Return type:

(Pandas dataframe)

parse_software(software_str: str) → List[Tuple[str, str]]

Processes the dependency and import information of each R package.

This includes: - Extracting the version information of each dependency/import if provided. - Providing the Item QID given the dependency/import label. - Creating a new Item if the dependency/import is not found in the

local knowledge graph.

Returns:

List of tuples including software QID and version.

Return type:

List[Tuple[str, str]]

parse_license(x: str) → List[Tuple[str, str]]

Splits string of licenses.

Takes into account that licenses are often not uniformly listed. Characters |, + and , are used to separate licenses. Further details on each license are often included in square brackets.

The concrete License is identified and linked to the corresponding item that has previously been imported from Wikidata. Further license information, when provided between round or square brackets, is added as a qualifier.

If a file license is mentioned, the linked to the file license in CRAN is added as a qualifier.

Parameters:

x (str) – String imported from CRAN representing license information.

Returns:

List of license tuples. Each tuple contains the license QID as the first element and the license qualifier as the second element.

Return type:

List[Tuple[str, str]]

page_entries(package_df) → List[Entry]

People named on the package page, for a package without database entries.

plan_people(linked: List[Tuple[str, str]]) → Tuple[List[Resolved], Resolved | None]

Resolved authors (Authors@R aut, or everyone in a free-text Author field) and maintainer.

apply_people() → None

Bring the author and maintainer statements in line with CRAN, only adding where possible.

Links already on the package stay; an author name string is dropped when that person is now linked as an item; a maintainer link is removed only when CRAN names another maintainer or ORPHANED. Without any people for the package (none read), nothing is changed. Labels and aliases of person items are never changed.

get_license_QID(license_str: str) → str

Returns the Wikidata item ID corresponding to a software license.

The same license is often denominated in CRAN using differents names. This function returns the wikidata item ID corresponding to a single unique license that is referenced in CRAN under different names (e.g. Artistic-2.0 and Artistic License 2.0 both refer to the same license, corresponding to item Q14624826).

Parameters:

license_str (str) – String corresponding to a license imported from CRAN.

Returns:

Wikidata item ID.

Return type:

(str)

get_wikidata_QID() → str | None

The Wikidata QID of the R package, from wikidata_r_packages() (read once per run).

get_versions()