Importer Module for zbMATH

mardi_importer.zbmath.ZBMathSource module

class mardi_importer.zbmath.ZBMathSource.ZBMathSource(*args, **kwargs)

Bases: ADataSource

Reads data from zb math API.

arxiv_exists(arxiv_id)

create_arxiv_item(publication, info_dict)

get_creation_date(xml_record)

Get creation date from xml record.

Parameters:

xml_record (xml element) – record returned by zbMath API

Returns:

creation date

Return type:

string

get_de_number(xml_record)

Get zbMath id from xml record.

Parameters:

xml_record (xml element) – record returned by zbMath API

Returns:

zbMath ID

Return type:

string

get_line(values)

get_updater(logger=None)

Return this source’s ZBMathUpdater, creating it once.

Sources are singletons, so the updater — and with it the author and journal caches — is shared by every task of a flow run in the same process.

old_process_data()

Overrides abstract method. Reads a raw zbMath data dump and processes it, then saves it as a csv.

old_write_data_dump()

Overrides abstract method. This method queries the zbMath API to get a data dump of all records, optionally between from_date and until_date

parse_record(xml_record)

Parse xml record from zbMath API.

Parameters:

xml_record (xml element) – record returned by zbMath API

Returns:

dict of (tag,value) pairs extracted from xml_record

Return type:

dict

process_data(resume_after_de=None, progress_callback=None, resume_offset=None, sync_every=1000)

Overrides abstract method. Reads a raw zbMath data dump and processes it, then saves it as a csv.

Every sync_every rows (and at the end) the file is made durable and progress_callback is called with the last written de_number; self.last_durable_offset then holds the file size up to and including that row. Passing it back as resume_offset together with resume_after_de truncates whatever was written after it, so a resumed conversion neither loses nor duplicates rows.

pull()

Pull data from the external source.

push(resume_after_de=None, progress_callback=None)

Updates the MaRDI Wikibase entities corresponding to zbMath publications. It creates a mardi_importer.zbmath.ZBMathPublication instance for each publication. Authors and journals are added, as well.

setup()

Create all necessary properties and entities for zbMath

update_from_diff(diff_path, mode='non_arxiv', logger=None, resume_after_de=None, progress_callback=None, missing_path=None, max_rows=None)

Apply a zbMATH diff file to the existing items.

Parameters:

• diff_path – Diff file built by mardi_importer.zbmath.diff.build_diff.

• mode – "non_arxiv" for publication items, "arxiv" for the separate arXiv items.

• resume_after_de – de_number to continue after.

• progress_callback – Called with each fully processed de_number.

• missing_path – Where diff rows whose item is not in the portal are collected (same format as the diff file). Without it, such a row raises.

• max_rows – Stop after this many rows (see ZBMathUpdater.apply).

Returns:

statistics of this pass.

Return type:

dict

update_references_from_diff(diff_path, logger=None, resume_after_de=None, progress_callback=None, missing_path=None, max_rows=None)

Merge changed reference lists from a diff file onto the items.

write_data_dump(start_after=0, output_path=None, progress_callback=None, resume_offset=None)

Overrides abstract method. This method queries the zbMath API to get a data dump of all records, optionally between from_date and until_date

progress_callback is called with the last downloaded id once that batch is on disk; self.last_durable_offset then holds the file size up to and including it. Passing that back as resume_offset (together with the id as start_after) truncates anything written after it.

write_subset_dump(file)

Overrides abstract method. This method queries the zbMath API to get a data dump of all records, optionally between from_date and until_date

mardi_importer.zbmath.misc module

mardi_importer.zbmath.misc.deduplicate_arxiv_file(old_arxiv_path, new_arxiv_path)

mardi_importer.zbmath.misc.get_info_from_doi(doi, key)

Query crossref API for DOI information.

Parameters:

• doi – doi

• key – document_title only for now

Returns:

document title

Return type:

title

mardi_importer.zbmath.misc.get_tag(tag_name, namespace)

Returns a fully qualified tag name.

Parameters:

• tag_name (string) – name of tag, e.g. author

• namespace (string) – namespace URL of a namespace

mardi_importer.zbmath.misc.parse_doi_info(val, work_info)

Function to extract information returned by a doi query for a specific tag.

Parameters:

• val (string) – tag, e.g. author

• work_info (dict) – information from doi query response

Returns:

information for specific tag, None if not found

Return type:

string

mardi_importer.zbmath.misc.run_references(dump_path, mc, log, resume_after_de=None, progress_callback=None, batch_size=100)

mardi_importer.zbmath.misc.search_item_by_property(property_id, value)

Search for pages in namespace 120 that have the statement:

haswbstatement:P<property_id>=<value>

Parameters:

value – e.g. “6369674”

Returns:

JSON response from the API

mardi_importer.zbmath.misc.split_file(processed_dump_path)

mardi_importer.zbmath.misc.year_to_str(value)

A publication year as an integer string, or "None".

process_data reads the raw dump with pandas in chunks, and a chunk in which any record lacks a year turns the whole year column into floats, so 2024 would come out as "2024.0". Coercing it here keeps the processed dump stable no matter which chunk a record lands in.