STU3 Python APIs¶

Bunsen offers Python APIs for PySpark users working with FHIR datasets. This includes basic functionality for working with FHIR Concept Maps, Bundles and Valuesets.

FHIR Bundles¶

Support for loading FHIR bundles into Bunsen. This includes the following features:

Allow users to load bundles from a given location
Convert bundle entries into Spark Dataframes
Save all entities with a bundle collection to a distinct table for each (e.g., an observation table, a condition table, and so on.)
Converts the results of a Bunsen query back into bundles that can then be used elsewhere.

See the methods below for details.

bunsen.stu3.bundles.extract_entry(sparkSession, javaRDD, resourceName)¶

Returns a dataset for the given entry type from the bundles.

Parameters:	sparkSession – the SparkSession instance javaRDD – the RDD produced by `load_from_directory()` or other methods in this package resourceName – the name of the FHIR resource to extract (condition, observation, etc)
Returns:	a DataFrame containing the given resource encoded into Spark columns

bunsen.stu3.bundles.from_json(df, column)¶

Takes a dataframe with JSON-encoded bundles in the given column and returns a Java RDD of Bundle records. Note this RDD contains Bundle records that aren’t serializable in Python, so users should use this class as merely a parameter to other methods in this module, like extract_entry.

Parameters:	df – a DataFrame containing bundles to decode column – the column in which the bundles to decode are stored
Returns:	a Java RDD of bundles for use with `extract_entry()`

bunsen.stu3.bundles.from_xml(df, column)¶

Takes a dataframe with XML-encoded bundles in the given column and returns a Java RDD of Bundle records. Note this RDD contains Bundle records that aren’t serializable in Python, so users should use this class as merely a parameter to other methods in this module, like extract_entry.

Parameters:	df – a DataFrame containing bundles to decode column – the column in which the bundles to decode are stored
Returns:	a Java RDD of bundles for use with `extract_entry()`

bunsen.stu3.bundles.load_from_directory(sparkSession, path, minPartitions=1)¶

Returns a Java RDD of bundles loaded from the given path. Note this RDD contains Bundle records that aren’t serializable in Python, so users should use this class as merely a parameter to other methods in this module, like extract_entry().

Parameters:	sparkSession – the SparkSession instance path – path to directory of FHIR bundles to load
Returns:	a Java RDD of bundles for use with `extract_entry()`

bunsen.stu3.bundles.save_as_database(sparkSession, path, databaseName, *resourceNames, **kwargs)¶

DEPRECATED. Users can easily do this by combining the load_from_directory and write_to_database functions.

Loads the bundles in the path and saves them to a database, where each table in the database has the same name of the resource it represents.

Parameters:	sparkSession – the SparkSession instance path – path to directory of FHIR bundles to load databaseName – name of the database to write the resources to resourceNames – the names of the FHIR resource to extract (condition, observation, etc)

bunsen.stu3.bundles.to_bundle(sparkSession, dataset)¶

Converts a dataset of FHIR resources to a bundle containing those resources. Use with caution against large datasets.

Parameters:	sparkSession – the SparkSession instance dataset – a DataFrame of encoded FHIR Resources
Returns:	a JSON bundle of the dataset contents

bunsen.stu3.bundles.write_to_database(sparkSession, javaRDD, databaseName, resourceNames)¶

Writes the bundles in the give RDD and saves them to a database, where each table in the database has the same name of the resource it represents.

Parameters:	sparkSession – the SparkSession instance javaRDD – the RDD produced by `load_from_directory()` or other methods in this package databaseName – name of the database to write the resources to resourceNames – the names of the FHIR resource to extract (condition, observation, etc)

FHIR Valuesets¶

Support for broadcasting valuesets and using them in user-defined functions in Spark queries.

bunsen.stu3.valuesets.get_current_valuesets(spark_session)¶

Returns the current valuesets in the same form that is accepted by the push_valuesets function above, that is the structure will follow this pattern: {referenceName: [(codeset, codevalue), (codeset, codevalue)]}

Parameters:	spark_session – the SparkSession instance
Returns:	a map containing the valuesets currently published to the cluster

bunsen.stu3.valuesets.isa_loinc(code_value, loinc_version=None)¶

Returns a hierarchy placeholder that will load all values that are descendents of a given LOINC code.

Parameters:	code_value – the parent code value loinc_version – the version of LOINC to use (uses latest if None is given)
Returns:	a placeholder for use with `push_valuesets()`

bunsen.stu3.valuesets.isa_snomed(code_value, snomed_version=None)¶

Returns a hierarchy placeholder that will load all values that are descendents of a given SNOMED code.

Parameters:	code_value – the parent code value loinc_version – the version of SNOMED to use (uses latest if None is given)
Returns:	a placeholder for use with `push_valuesets()`

bunsen.stu3.valuesets.pop_valuesets(spark_session)¶

Pops the current valuesets from the stack, returning true if there remains an active valueset, or false otherwise.

Parameters:	spark_session – the SparkSession instance

bunsen.stu3.valuesets.push_valuesets(spark_session, valueset_map, database='ontologies')¶

Pushes valuesets onto a stack and registers an in_valueset user-defined function that uses this content.

The valueset_map takes the form of {referenceName: [(codeset, codevalue), (codeset, codevalue)]} to specify which codesets/values are used for the given valueset reference name.

Rather than explicitly passing a list of (codeset, codevalue) tuples, users may instead load particular value sets or particular hierarchies by providing a ValueSetPlaceholder or HierarchyPlaceholder that instructs the system to load codes belonging to a particular value set or hierarchical system, respectively. See the isa_loinc and isa_snomed functions above for details.

Finally, ontology information is assumed to be stored in the ‘ontologies’ database by default, but users can specify another database name if they have customized ontologies that are separated from the default ontologies database.

Parameters:	spark_session – the SparkSession instance valueset_map – a map containing value set structures to publish database – the database from which value set data is loaded

bunsen.stu3.valuesets.valueset(valueset_uri, valueset_version)¶

Creates a placeholder specifying a specific valueset for use with push_valuesets().

Parameters:	valueset_uri – the URI of the valueset valueset_version – the version of the valueset
Returns:	a placeholder for use with `push_valuesets()`

APIS for Loading ValueSets and ConceptMaps¶

Bunsen Python API for working with Code Systems.

bunsen.stu3.codes.create_concept_maps(spark_session)¶

Creates a new, empty bunsen.codes.ConceptMaps instance.

Returns:	an empty `bunsen.codes.ConceptMaps` instance

bunsen.stu3.codes.create_hierarchies(spark_session)¶

Creates a new, empty bunsen.codes.Hierarchies instance.

Returns:	an empty `bunsen.codes.Hierarchies` instance

bunsen.stu3.codes.create_value_sets(spark_session)¶

Creates a new, empty bunsen.codes.ValueSets instance.

Returns:	an empty `bunsen.codes.ValueSets` instance

bunsen.stu3.codes.get_concept_maps(spark_session, database='ontologies')¶

Returns a bunsen.codes.ConceptMaps instance for the given database.

Parameters:	database – the database containing the concept maps to load
Returns:	a `bunsen.codes.ConceptMaps` with the loaded maps

bunsen.stu3.codes.get_hierarchies(spark_session, database='ontologies')¶

Returns a bunsen.codes.Hierarchies instance for the given database.

Parameters:	database – the database containing the hierarchies to load
Returns:	a `bunsen.codes.Hierarchies` with the loaded value sets

bunsen.stu3.codes.get_value_sets(spark_session, database='ontologies')¶

Returns a bunsen.codes.ValueSets instance for the given database.

Parameters:	database – the database containing the value sets to load
Returns:	a `bunsen.codes.ValueSets` with the loaded value sets

Core library for working with Concept Maps and Value Sets, and hierarchical code systems in Bunsen. See the ConceptMaps class, ValueSets class, and Hierarchies class for details.

class bunsen.codes.ConceptMaps(spark_session, jconcept_maps, jfunctions, java_package)¶

An immutable collection of FHIR Concept Maps to be used to map value sets. These instances are typically created via the :py:module bunsen.codes.stu3

add_mappings(url, version, mappings)¶

Returns a new ConceptMaps instance with the given mappings added to an existing map. The mappings parameter must be a list of tuples of the form [(source_system, source_value, target_system, target_value, equivalence)].

Parameters:	url – URL of the ConceptMap to add mappings to version – Version of the ConceptMap to add mappings to mappings – A list of tuples representing the mappings to add
Returns:	a `ConceptMaps` instance with the added mappings

get_map_as_xml(url, version)¶

Returns an XML string containing the specified concept map.

Parameters:	url – URL of the ConceptMap to return version – Version of the ConceptMap to return
Returns:	a string containing the ConceptMap in XML form

get_mappings(url=None, version=None)¶

Returns a dataset of all mappings which may be filtered by an optional concept map url and concept map version.

Parameters:	url – Optional URL of the mappings to return version – Optional version of the mappings to return
Returns:	a DataFrame of mapping records

get_maps()¶

Returns a dataset of FHIR ConceptMaps without the nested mapping content, allowing users to explore mapping metadata.

The mappings themselves are excluded because they can become quite large, so users should use the get_mappings method to explore a table of them.

Returns:	a DataFrame of FHIR ConceptMap resources managed by this object

latest_version(url)¶

Returns the latest version of a map, or None if there is none.”

Parameters:	url – the URL identifying a given concept map
Returns:	the version of the given map

with_disjoint_maps_from_directory(path, database='ontologies')¶

Returns a new ConceptMaps instance with all value sets read from the given directory path that are disjoint with value sets stored in the given database. The directory may be anything readable from a Spark path, including local filesystems, HDFS, S3, or others.

Parameters:	path – Path to directory containing FHIR ConceptMap resources database – The database in which existing concept maps are stored
Returns:	a `ConceptMaps` instance with the added maps

with_maps_from_directory(path)¶

Returns a new ConceptMaps instance with all maps read from the given directory path. The directory may be anything readable from a Spark path, including local filesystems, HDFS, S3, or others.

Parameters:	path – Path to directory containing FHIR ConceptMap resources
Returns:	a `ConceptMaps` instance with the added maps

with_new_map(url, version, source, target, experimental=True, mappings=[])¶

Returns a new ConceptMaps instance with the given map added. Callers may include a list of mappings tuples in the form of [(source_system, source_value, target_system, target_value, equivalence)].

Parameters:	url – URL of the ConceptMap to add version – Version of the ConceptMap to add source – source URI of the ConceptMap target – target URI of the ConceptMap experimental – a Boolean variable indicating whether the map should be labeled as experimental mappings – A list of tuples representing the mappings to add
Returns:	a `ConceptMaps` instance with the added map

write_to_database(database)¶

Writes the mapping content to the given database, creating a mappings and conceptmaps table if they don’t exist.

Parameters:	database – the database to write the concept maps to

class bunsen.codes.Hierarchies(spark_session, jhierarchies)¶

An immutable collection of values from hierarchical code systems to be used for ontologically-based queries.

get_ancestors(url=None, version=None)¶

Returns a dataset of ancestor values representing the transitive closure of codes in this Hierarchies instance filtered by an optional hierarchy uri and version.

Parameters:	url – Optional URL of hierarchy to return version – Optional version of the hierarchy to return
Returns:	a DataFrame of ancestor records

latest_version(uri)¶

Returns the latest version of a hierarchy, or None if there is none.

Parameters:	uri – URI of the concept hierarchy to return
Returns:	the version of the hierarchy, or None if there is none

write_to_database(database)¶

Write the ancestor content to the given database, create an ancestors table if they don’t exist.

Parameters:	database – the database to write the hierarchies to

class bunsen.codes.ValueSets(spark_session, jvalue_sets, jfunctions, java_package)¶

An immutable collection of FHIR Value Sets to be used to for ontologically-based queries.

add_values(url, version, values)¶

Returns a new ValueSets instance with the given values added to an existing value set. The values parameter must be a list of the form [(sytem, value)].

Parameters:	url – URL of the ValueSet to add values to version – Version of the ValueSet to add values to mappings – A list of tuples representing the values to add
Returns:	a `ValueSets` instance with the added values

get_value_set_as_xml(url, version)¶

Returns an XML string containing the specified value set.

Parameters:	url – URL of the ValueSet to return version – Version of the ValueSet to return
Returns:	a string containing the ValueSet in XML form

get_value_sets()¶

Returns a dataset of FHIR ValueSets without the nested value content, allowing users to explore value set metadata.

The values themselves are excluded because they can be become quite large, so users should use the get_values method to explore them.

Returns:	a dataframe of FHIR ValueSets

get_values(url=None, version=None)¶

Returns a dataset of all values which may be filtered by an optional value set url and value set version.

Parameters:	url – Optional URL of ValueSet to return version – Optional version of the ValueSet to return
Returns:	a DataFrame of values

latest_version(url)¶

Returns the latest version of a value set, or None if there is none.

Parameters:	url – URL of the ValueSet to return
Returns:	the version of the ValueSet, or None if there is none

with_disjoint_value_sets_from_directory(path, database='ontologies')¶

Returns a new ValueSets instance with all value sets read from the given directory path that are disjoint with value sets stored in the given database. The directory may be anything readable from a Spark path, including local filesystems, HDFS, S3, or others.

Parameters:	path – Path to directory containing FHIR ValueSet resources database – The database in which existing value sets are stored
Returns:	a `ValueSets` instance with the added value sets

with_new_value_set(url, version, experimental=True, values=[])¶

Returns a new ValueSets instance with the given value set added. Callers may include a list of value tuples in the form of [(system, value)].

Parameters:	url – URL of the ValueSet to add version – Version of the ValueSet to add experimental – a Boolean variable indicating whether the ValueSet should be labeled as experimental values – A list of tuples representing the values to add
Returns:	a `ValueSets` instance with the added value set.

with_value_sets(df)¶

Returns a new ValueSets instance that includes the ValueSet FHIR resources encoded in the given Spark DataFrame.

Parameters:	df – A Spark DataFrame containing the valueset FHIR resource
Returns:	a `ValueSets` instance with the added value sets

with_value_sets_from_directory(path)¶

Returns a new ValueSets instance with all value sets read from the given directory path. The directory may be anything readable from a Spark path, including local filesystems, HDFS, S3, or others.

Parameters:	path – Path to directory containing FHIR ValueSet resources
Returns:	a `ValueSets` instance with the added value sets

write_to_database(database)¶

Writes the value set content to the given database, creating a values and valuesets table if they don’t exist.

Parameters:	database – the database to write the value sets to