com.cerner.bunsen.spark.codes.base

Class AbstractConceptMaps<T extends org.hl7.fhir.instance.model.api.IBaseResource,C extends AbstractConceptMaps<T,C>>

java.lang.Object

com.cerner.bunsen.spark.codes.base.AbstractConceptMaps<T,C>

Type Parameters:

T - the type of the FHIR ConceptMap objects being used

C - the type of the subclass of this class being used.

Direct Known Subclasses:

ConceptMaps

public abstract class AbstractConceptMaps<T extends org.hl7.fhir.instance.model.api.IBaseResource,C extends AbstractConceptMaps<T,C>>
extends Object

This is a partial implementation of logic to manage FHIR ConceptMaps. It is designed to encapsulate as much functionality as possible while remaining independent of specific FHIR versions. Users should generally not use this class directly, but rather consume the subclass that corresponds to the FHIR version they are using.

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	CONCEPT_MAP_TABLE Defalt table name where concept maps are stored.
protected SparkRowConverter	conceptMapRowConverter
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	conceptMaps
protected ca.uhn.fhir.context.FhirVersionEnum	fhirVersion
static String	MAPPING_DATABASE Default database name where mapping information is stored.
protected static org.apache.spark.sql.Encoder<Mapping>	MAPPING_ENCODER An encoder for serializing mappings.
static String	MAPPING_TABLE Default table name where expanded mapping information is stored.
protected org.apache.spark.sql.Dataset<Mapping>	mappings
protected org.apache.spark.sql.Dataset<UrlAndVersion>	members
protected org.apache.spark.sql.SparkSession	spark
protected static Pattern	TABLE_NAME_PATTERN
protected static org.apache.spark.sql.Encoder<UrlAndVersion>	URL_AND_VERSION_ENCODER

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractConceptMaps(org.apache.spark.sql.SparkSession spark, ca.uhn.fhir.context.FhirVersionEnum fhirVersion, org.apache.spark.sql.Dataset<UrlAndVersion> members, org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> conceptMaps, org.apache.spark.sql.Dataset<Mapping> mappings, SparkRowConverter conceptMapRowConverter)

Method Summary

Modifier and Type	Method and Description
protected abstract void	addToConceptMap(T map, org.apache.spark.sql.Dataset<Mapping> mappings) Adds the given mappings to the concept map.
abstract org.apache.spark.broadcast.Broadcast<BroadcastableMappings>	broadcast(Map<String,String> conceptMapUriToVersion) Broadcast mappings stored in the given conceptMaps instance that match the given conceptMapUris.
org.apache.spark.broadcast.Broadcast<BroadcastableMappings>	broadcast(Set<String> conceptMapUris, boolean includeExperimental) Broadcast mappings stored in the given conceptMaps instance that match the given conceptMapUris.
T	getConceptMap(String uri, String version) Returns the concept map with the given uri and version, or null if there is no such map.
org.apache.spark.sql.Dataset<Mapping>	getLatestMappings(Set<String> uris, boolean includeExperimental) Returns a dataset with the latest mappings for each valueset here.
Map<String,String>	getLatestVersions(boolean includeExperimental) Returns the latest versions of all concept maps.
Map<String,String>	getLatestVersions(Set<String> urls, boolean includeExperimental) Returns the latest versions of a given set of concept maps.
org.apache.spark.sql.Dataset<Mapping>	getMappings() Returns a dataset of all mappings in this collection.
org.apache.spark.sql.Dataset<Mapping>	getMappings(Map<String,String> uriToVersion) Returns a dataset with the mappings for each uri and version.
org.apache.spark.sql.Dataset<Mapping>	getMappings(String uri, String version) Returns the mappings for the given URI and version.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	getMaps() Returns a dataset of concept maps to inspect metadata.
static org.apache.spark.sql.Encoder<UrlAndVersion>	getUrlAndVersionEncoder() Returns the encoder for UrlAndVersion tuples.
protected org.apache.spark.sql.Dataset<UrlAndVersion>	getUrlAndVersions(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> conceptMaps) Returns a simple dataset of URL and versions of concept maps.
protected boolean	hasDuplicateUrlAndVersions(org.apache.spark.sql.Dataset<UrlAndVersion> membersToCheck) Returns true if the UrlAndVersions of new value sets contains duplicates with the current ValueSets.
protected abstract C	newInstance(org.apache.spark.sql.SparkSession spark, org.apache.spark.sql.Dataset<UrlAndVersion> members, org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> conceptMaps, org.apache.spark.sql.Dataset<Mapping> mappings) Returns a new instance of this ConceptMaps type with the given parameters.
abstract C	withConceptMaps(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> conceptMaps) Returns a new ConceptMaps instance that includes the given maps.
protected C	withConceptMaps(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> newMaps, org.apache.spark.sql.Dataset<Mapping> newMappings)
C	withConceptMaps(List<T> conceptMaps) Returns a new ConceptMaps instance that includes the given maps.
C	withConceptMaps(T... conceptMap) Returns a new ConceptMaps instance that includes the given map.
C	withDisjointMapsFromDirectory(String path) Returns all concept maps that are disjoint with concept maps stored in the default database and adds them to our collection.
C	withDisjointMapsFromDirectory(String path, String database) Returns all concept maps that are disjoint with concept maps stored in the default database and adds them to our collection.
C	withExpandedMap(T conceptMap, org.apache.spark.sql.Dataset<Mapping> mappings) Returns a new ConceptMaps instance that includes the given map and expanded mappings.
C	withMapsFromDirectory(String path) Reads all concept maps from a given directory and adds them to our collection.
void	writeToDatabase(String database) Writes the updated concept maps to a database using the default "mappings" and "conceptmaps" table names.
void	writeToTables(String mappingsTable, String conceptMapTable) Writes mappings to the given tables.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAPPING_ENCODER

protected static final org.apache.spark.sql.Encoder<Mapping> MAPPING_ENCODER

An encoder for serializing mappings.

URL_AND_VERSION_ENCODER

protected static final org.apache.spark.sql.Encoder<UrlAndVersion> URL_AND_VERSION_ENCODER

MAPPING_DATABASE

public static final String MAPPING_DATABASE

Default database name where mapping information is stored.

See Also:

Constant Field Values

MAPPING_TABLE

public static final String MAPPING_TABLE

Default table name where expanded mapping information is stored.

See Also:

Constant Field Values

CONCEPT_MAP_TABLE

protected static final String CONCEPT_MAP_TABLE

Defalt table name where concept maps are stored.

See Also:

Constant Field Values

TABLE_NAME_PATTERN

protected static final Pattern TABLE_NAME_PATTERN

spark

protected final org.apache.spark.sql.SparkSession spark

fhirVersion

protected final ca.uhn.fhir.context.FhirVersionEnum fhirVersion

members

protected final org.apache.spark.sql.Dataset<UrlAndVersion> members

conceptMaps

protected final org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> conceptMaps

mappings

protected final org.apache.spark.sql.Dataset<Mapping> mappings

conceptMapRowConverter

protected final SparkRowConverter conceptMapRowConverter

Constructor Detail

AbstractConceptMaps

protected AbstractConceptMaps(org.apache.spark.sql.SparkSession spark,
                              ca.uhn.fhir.context.FhirVersionEnum fhirVersion,
                              org.apache.spark.sql.Dataset<UrlAndVersion> members,
                              org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> conceptMaps,
                              org.apache.spark.sql.Dataset<Mapping> mappings,
                              SparkRowConverter conceptMapRowConverter)

Method Detail

getUrlAndVersionEncoder

public static org.apache.spark.sql.Encoder<UrlAndVersion> getUrlAndVersionEncoder()

Returns the encoder for UrlAndVersion tuples.

Returns:

an encoder for UrlAndVersion tuples.

getUrlAndVersions

protected org.apache.spark.sql.Dataset<UrlAndVersion> getUrlAndVersions(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> conceptMaps)

Returns a simple dataset of URL and versions of concept maps.

Parameters:

conceptMaps - a datafrae of concept maps to scan.

Returns:

a dataframe of URL and version details for the maps.

withDisjointMapsFromDirectory

public C withDisjointMapsFromDirectory(String path)

Returns all concept maps that are disjoint with concept maps stored in the default database and adds them to our collection. The directory may be anything readable from a Spark path, including local filesystems, HDFS, S3, or others.

Parameters:

path - a path from which disjoint concept maps will be loaded

Returns:

an instance of ConceptMaps that includes content from that directory that is disjoint with content already contained in the default database.

withDisjointMapsFromDirectory

public C withDisjointMapsFromDirectory(String path,
                                       String database)

Returns all concept maps that are disjoint with concept maps stored in the default database and adds them to our collection. The directory may be anything readable from a Spark path, including local filesystems, HDFS, S3, or others.

Parameters:

path - a path from which disjoint concept maps will be loaded

database - the database to check concept maps against

Returns:

an instance of ConceptMaps that includes content from that directory that is disjoint with content already contained in the default database.

withMapsFromDirectory

public C withMapsFromDirectory(String path)

Reads all concept maps from a given directory and adds them to our collection. The directory may be anything readable from a Spark path, including local filesystems, HDFS, S3, or others.

Parameters:

path - a path from which concept maps will be loaded

Returns:

a instance of ConceptMaps that includes the contents from that directory.

withConceptMaps

public abstract C withConceptMaps(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> conceptMaps)

Returns a new ConceptMaps instance that includes the given maps.

Parameters:

conceptMaps - concept maps to add to the returned collection.

Returns:

a new ConceptMaps instance with the values added.

withConceptMaps

public C withConceptMaps(T... conceptMap)

Returns a new ConceptMaps instance that includes the given map.

Parameters:

conceptMap - concept maps to add

Returns:

a new ConceptMaps instance with the values added.

withConceptMaps

public C withConceptMaps(List<T> conceptMaps)

Returns a new ConceptMaps instance that includes the given maps.

Parameters:

conceptMaps - concept maps to add

Returns:

a new ConceptMaps instance with the values added.

withConceptMaps

protected C withConceptMaps(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> newMaps,
                            org.apache.spark.sql.Dataset<Mapping> newMappings)

newInstance

protected abstract C newInstance(org.apache.spark.sql.SparkSession spark,
                                 org.apache.spark.sql.Dataset<UrlAndVersion> members,
                                 org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> conceptMaps,
                                 org.apache.spark.sql.Dataset<Mapping> mappings)

Returns a new instance of this ConceptMaps type with the given parameters.

Parameters:

spark - the spark session

members - Concept map URL and version details

conceptMaps - the concept maps to include

mappings - the mappings themselves

Returns:

a new ConceptMaps instance.

withExpandedMap

public C withExpandedMap(T conceptMap,
                         org.apache.spark.sql.Dataset<Mapping> mappings)

Returns a new ConceptMaps instance that includes the given map and expanded mappings. This method is convenient when mappings themselves are loaded from some ETL operation that produces them.

Parameters:

conceptMap - concept map to add

mappings - dataset of mappings to add to add

Returns:

a new ConceptMaps instance with the values added.

addToConceptMap

protected abstract void addToConceptMap(T map,
                                        org.apache.spark.sql.Dataset<Mapping> mappings)

Adds the given mappings to the concept map.

Parameters:

map - the concept map

mappings - the mappings to add

getConceptMap

public T getConceptMap(String uri,
                       String version)

Returns the concept map with the given uri and version, or null if there is no such map.

Parameters:

uri - the uri of the map to return

version - the version of the map to return

Returns:

the specified concept map.

getMaps

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> getMaps()

Returns a dataset of concept maps to inspect metadata. Since the mappings themselves can be quite large, the maps in this dataset do not contain them. Instead, users should use the getMappings() method to query mappings in depth.

Returns:

a dataset of concept maps that do not contain mappings.

getMappings

public org.apache.spark.sql.Dataset<Mapping> getMappings()

Returns a dataset of all mappings in this collection. This is generally used for inspection and debugging of mappings.

Returns:

a dataset of all mappings.

getMappings

public org.apache.spark.sql.Dataset<Mapping> getMappings(String uri,
                                                         String version)

Returns the mappings for the given URI and version.

Parameters:

uri - the uri of the concept map for which we get mappings

version - the version of the concept map for which we get mappings

Returns:

a dataset of mappings for the given URI and version.

getMappings

public org.apache.spark.sql.Dataset<Mapping> getMappings(Map<String,String> uriToVersion)

Returns a dataset with the mappings for each uri and version.

Parameters:

uriToVersion - a map of concept map URI to the version to load

Returns:

a dataset of mappings for the given URIs and versions.

getLatestMappings

public org.apache.spark.sql.Dataset<Mapping> getLatestMappings(Set<String> uris,
                                                               boolean includeExperimental)

Returns a dataset with the latest mappings for each valueset here.

Parameters:

uris - URIs for the value sets

includeExperimental - include valuesets labeled as experimental

Returns:

a dataset of the latest mappings for them.

getLatestVersions

public Map<String,String> getLatestVersions(boolean includeExperimental)

Returns the latest versions of all concept maps.

Parameters:

includeExperimental - flag to include concept maps marked as experimental

Returns:

a map of concept map URLs to the latest version for them.

getLatestVersions

public Map<String,String> getLatestVersions(Set<String> urls,
                                            boolean includeExperimental)

Returns the latest versions of a given set of concept maps.

Parameters:

urls - a set of URLs to retrieve the latest version for, or null to load them all.

includeExperimental - flag to include concept maps marked as experimental

Returns:

a map of concept map URLs to the latest version for them.

hasDuplicateUrlAndVersions

protected boolean hasDuplicateUrlAndVersions(org.apache.spark.sql.Dataset<UrlAndVersion> membersToCheck)

Returns true if the UrlAndVersions of new value sets contains duplicates with the current ValueSets.

Parameters:

membersToCheck - url and versions to check for duplciates

Returns:

true if there are duplicates, false otherwise.

writeToDatabase

public void writeToDatabase(String database)

Writes the updated concept maps to a database using the default "mappings" and "conceptmaps" table names.

Parameters:

database - the database name to which the concepts are saved.

writeToTables

public void writeToTables(String mappingsTable,
                          String conceptMapTable)

Writes mappings to the given tables.

Warning: these updates are likely not atomic due to the lack of transactional semantics in the underlying data store. Concurrent users may see previous items removed before new ones are added, or items appear separately than others. This is intended for use in a user-specific sandbox or staging environment.

Parameters:

mappingsTable - name of the table containing the mapping records

conceptMapTable - name of the table containing the concept map metadata

broadcast

public org.apache.spark.broadcast.Broadcast<BroadcastableMappings> broadcast(Set<String> conceptMapUris,
                                                                             boolean includeExperimental)

Broadcast mappings stored in the given conceptMaps instance that match the given conceptMapUris.

Parameters:

conceptMapUris - the URIs to broadcast.

includeExperimental - flag to include experimental map versions in the broadcast.

Returns:

a broadcast variable containing a mappings object usable in UDFs.

broadcast

public abstract org.apache.spark.broadcast.Broadcast<BroadcastableMappings> broadcast(Map<String,String> conceptMapUriToVersion)

Broadcast mappings stored in the given conceptMaps instance that match the given conceptMapUris.

Parameters:

conceptMapUriToVersion - map of the concept map URIs to broadcast to their versions.

Returns:

a broadcast variable containing a mappings object usable in UDFs.