Importer Module for Julia

This module imports Julia packages for numerical analysis from the Julia General registry, the community registry of Julia packages (MIT-licensed). The registry plays the role CRAN plays for R; a package’s own repository supplies the rest.

Scope

Packages whose repository is under one of the GitHub organisations SciML, JuliaMath, JuliaLinearAlgebra, JuliaNLSolvers or JuliaDiff. Binary wrappers (_jll packages) are excluded, and so are packages registered in a sub-directory of a shared repository (internal splits such as the OrdinaryDiffEq* solver packages), except StochasticDiffEq and DelayDiffEq. About 290 packages.

What is imported

From the registry, pinned to the registry commit:

Name:: As the identifier Julia General registry package name (without .jl), linked to the package’s page on JuliaHub.
Repository:: source code repository.
Version:: The latest non-yanked version, as software version identifier.

From the package’s repository, pinned to a commit:

License:: Read from the licence file. The licence named at the start of the file decides; bundled third-party notices further down are ignored. A package whose licence cannot be determined is not written.
Authors:: From Project.toml and the software authors of CITATION.cff. A person with an item becomes author, with the name as stated in object named as; anyone else an author name string.
Dependencies:: depends on software, to packages that exist in the knowledge graph or are imported in the same run.
Publications:: DOIs and arXiv ids from CITATION.bib/CITATION.cff become described by source. Zenodo DOIs are the software’s own releases and are skipped.

Every statement carries a reference: stated in the Julia General registry for registry facts, the reference URL of the file it was read from, and the retrieved date.

Matching existing items

An existing item is updated only when it is the same software: it already carries the package’s registry name, or it records the package’s own repository, or it has the package’s name and a repository named <Name>.jl (the same package before its repository moved organisation). A name shared with different software — SUNDIALS and Sundials.jl, the FFTW C library and FFTW.jl — creates a new item and leaves the existing one untouched. Updates only add statements; nothing is overwritten or removed.

People

An existing person item is reused only when it is certainly the same person: it carries the person’s ORCID, or it is the same-named author of a paper the person’s own package cites. Otherwise a person who can be identified — the same e-mail address across packages, a co-author of a cited paper being created, or an ORCID — gets a new item; similarly named existing items are logged as possible duplicates for manual merging. E-mail addresses are used only to group mentions of the same person and are never written; every written value is checked for one.

Publications

A cited DOI or arXiv id already in the knowledge graph is linked. Otherwise the journal DOI of an arXiv preprint and the exact title are tried, and only then is the publication created, through the Crossref or arXiv module.

Usage

mardi-importer import-julia --dry-run            # decide and report, write nothing
mardi-importer import-julia                      # full import of every package in scope
mardi-importer import-julia --packages Optim NLsolve

Credentials are read from JULIA_USER / JULIA_PASS; publications use the Crossref and arXiv credentials. Re-running is safe: packages are found again by their registry name and only new information is added.

mardi_importer.julia.JuliaSource class

Julia packages for numerical analysis, from the Julia General registry.

Scope: packages whose repository is under SciML, JuliaMath, JuliaLinearAlgebra, JuliaNLSolvers or JuliaDiff (no _jll binary wrappers, no sub-directory packages except StochasticDiffEq and DelayDiffEq). For each package the importer writes the registry facts (name, repository, every registered version with the day it was registered), the licence, authors, dependencies and the publications its citation file names — every statement with a reference to where it was read.

Decisions encoded here (recorded in the MaRDI agents project, D016–D024):

Packages. An existing item is updated only when it is the same software: it already carries this package’s registry name; or it records the package’s own repository; or it has the package’s name and a repository named <Name>.jl (the same package before its repository moved). A name shared with other software (SUNDIALS vs. Sundials.jl) creates a new item and leaves the existing one untouched. Updates only add, never overwrite.
People. An existing person item is reused only when it is certainly the same person: it carries the person’s ORCID, or it is the same-named author of a paper the person’s own package cites. Otherwise an identifiable person — joined by e-mail across packages, co-author of a cited paper being created, or holding an ORCID — gets a new item; similarly named items are logged as possible duplicates, never merged. Everyone else stays an author name string. E-mail addresses are used to join mentions only and are never written.
Publications. A cited DOI or arXiv id already in MaRDI is linked (to every copy, if MaRDI holds duplicates); otherwise the journal DOI of an arXiv preprint or an exact title is tried; otherwise the publication is created through the Crossref or arXiv source. Zenodo DOIs are the software’s own releases and are not treated as publications. Packages link to publications with described by source.
Licence. A package whose licence cannot be determined is not written.

mardi_importer.julia.JuliaSource.same_software(pkg: dict, by_repo: set[str], by_name: set[str], urls_of: dict[str, list[str]]) → list[str][source]

Existing items that are this package, best evidence first. Empty → create.

the item records the package’s registered repository and the package owns it (a sub-directory package shares it with its parent, so the name must agree too — DelayDiffEq is not OrdinaryDiffEq.jl);
the item has the package’s name and a repository URL named <Name>.jl.

A shared name alone is never enough.

mardi_importer.julia.JuliaSource.decide_person(person: Person, by_orcid: dict[str, list[str]], coauthor: dict[str, str]) → None[source]: Reuse an item only when certain; otherwise create one for identifiable people.

mardi_importer.julia.JuliaSource.norm_title(title: str) → str[source]

class mardi_importer.julia.JuliaSource.JuliaSource(*args, **kwargs)[source]

Bases: ADataSource

Imports Julia numerical-analysis packages from the General registry.

setup()[source]: Import the Wikidata entities used, and create the local ones.

pull(names: Iterable[str] | None = None, registry_path: str | None = None, orcid_links: str | None = None) → list[JuliaPackage][source]

Read the registry and the packages’ repositories, and decide every write.

Parameters:

names – Restrict to these package names (all in scope if omitted).
registry_path – An existing clone of General (cloned if omitted).
orcid_links – Optional CSV of ORCIDs found by an e-mail search (columns spellings, packages, orcid, name_agrees); used by name and package, never by address.

push() → dict[source]: Write people, then publications, then packages, then dependencies.

summary(results: dict | None = None) → dict[source]: What was (or, before push, would be) written. Contains no e-mail address.

mardi_importer.julia.JuliaPackage class

One Julia package: the statements it should have, and writing them.

Statements are first planned as plain data (Planned) so the rules can be tested without a Wikibase; JuliaPackage.write() then turns them into claims.

Every statement carries a reference — stated in the Julia General registry for registry facts, reference URL of the exact file it was read from (pinned to a commit), and retrieved. Properties and items are given as Wikidata IDs and resolved to local IDs by mardiclient.

Every registered (non-yanked) version becomes a software version identifier statement qualified with its publication date, the day the version was registered in General, as the CRAN source does for R packages; its reference URL is the registering commit.

An existing item is only ever added to (never overwritten or pruned): a value already present is skipped, and a single-valued property that already holds a different value is left alone and reported as a conflict.

class mardi_importer.julia.JuliaPackage.JuliaPackage(name: 'str', uuid: 'str', repo: 'str', path: 'str', registry_sha: 'str', retrieved: 'str', metadata: 'RepoMetadata', subdir: 'str | None' = None, version: 'str | None' = None, versions: 'list[tuple[str, str | None, str | None]]'=<factory>, action: 'str' = 'create', qid: 'str | None' = None, matched_by: 'str | None' = None, wikidata_qid: 'str | None' = None, wikidata_retrieved: 'str | None' = None, papers: 'list[str]' = <factory>, authors: 'list[tuple[Mention, Person]]'=<factory>, conflicts: 'list[dict]' = <factory>, notes: 'list[str]' = <factory>)[source]

Bases: object

action: str = 'create'

authors: list[tuple[Mention, Person]]

conflicts: list[dict]

property label: str

matched_by: str | None = None

static merge(planned: list[Planned], existing: dict[str, list[str]]) → tuple[list[Planned], list[dict]][source]: Add-only: drop what is already there; keep single-valued conflicts out.

metadata: RepoMetadata

name: str

notes: list[str]

papers: list[str]

path: str

plan(julia_package_class: str) → list[Planned][source]: Every statement for this package except dependencies (planned later).

plan_authors() → list[Planned][source]

One statement per person: an item where the person has one, else a name string.

The name as the source states it is kept in object named as.

plan_dependencies(qid_of: dict[str, str]) → list[Planned][source]: depends on software to packages that exist or were created in this run.

qid: str | None = None

registry_sha: str

repo: str

retrieved: str

subdir: str | None = None

uuid: str

version: str | None = None

versions: list[tuple[str, str | None, str | None]]

wikidata_qid: str | None = None

wikidata_retrieved: str | None = None

write(api, planned: list[Planned], registry_item: str) → str | None[source]: Create the item, or add the planned statements to the existing one.

class mardi_importer.julia.JuliaPackage.Planned(prop: str, value: Any, ref_url: str | None, retrieved: str | None, stated_in_registry: bool = False, qualifiers: list[tuple] = <factory>)[source]

Bases: object

One statement to write: property, value, qualifiers, and its reference.

A qualifier is (property, value) or (property, value, claim kwargs), the kwargs going to get_claim (e.g. a time precision).

prop: str

qualifiers: list[tuple]

ref_url: str | None

property referenced: bool

retrieved: str | None

stated_in_registry: bool = False

value: Any

mardi_importer.julia.JuliaPackage.add_planned(api, item, st: Planned, registry_item: str) → None[source]: Add one planned statement with its qualifiers and reference.

mardi_importer.julia.JuliaPackage.existing_values(api, item, planned: list[Planned]) → dict[str, list[str]][source]

Current values of the planned properties, including URL values.

MardiItem.get_value skips URL-typed properties, which would make an existing repository look absent and be added twice.

A version without its publication date counts as absent, so that planning it again completes the existing statement (append-or-replace keeps the value and adds the qualifier) instead of skipping it.

mardi_importer.julia.JuliaPackage.resolve_items(api, planned: list[Planned]) → list[Planned][source]: Replace wd:Q… values by local QIDs, so they compare with existing claims.

mardi_importer.julia.JuliaPackage.time_value(day: str) → str[source]