Source Name
Integration Details
Concept Mapping
This ingestion source maps the following Source System Concepts to DataHub Concepts:
Source Concept | DataHub Concept | Notes |
---|---|---|
Data Platform | ||
Dataset | ||
Data Job | ||
Data Flow | ||
Chart | ||
Dashboard | ||
User (a.k.a CorpUser) | ||
CorpGroup | ||
Domain | ||
Container | ||
Tag | ||
GlossaryTerm | ||
GlossaryNode | ||
Assertion | ||
DataProcess | ||
MlFeature | ||
MlFeatureTable | ||
MlModel | ||
MlModelDeployment | ||
MlPrimaryKey | ||
SchemaField | ||
DataHubPolicy | ||
DataHubIngestionSource | ||
DataHubSecret | ||
DataHubExecutionRequest | ||
DataHubREtention |
Supported Capabilities
Capability | Status | Notes |
---|---|---|
Data Container | ✅ | Enabled by default |
Detect Deleted Entities | ✅ | Requires recipe configuration |
Data Domain | ❌ | Requires transformer |
Dataset Profiling | ✅ | Requires acryl-datahub[source-usage-name] |
Dataset Usage | ✅ | Requires acryl-datahub[source-usage-name] |
Extract Descriptions | ✅ | Enabled by default |
Extract Lineage | ✅ | Enabled by default |
Extract Ownership | ✅ | Enabled by default |
Extract Tags | ❌ | Requires transformer |
Partition Support | ❌ | Not applicable to source |
Platform Instance | ❌ | Not applicable to source |
... |
Metadata Ingestion Quickstart
Prerequisites
In order to ingest metadata from [Source Name], you will need:
- eg. Python version, source version, source access requirements
- eg. Steps to configure source access
- ...
Install the Plugin(s)
Run the following commands to install the relevant plugin(s):
pip install 'acryl-datahub[source-name]'
pip install 'acryl-datahub[source-usage-name]'
Configure the Ingestion Recipe(s)
Use the following recipe(s) to get started with ingestion.
For general pointers on writing and running a recipe, see our main recipe guide.
'acryl-datahub[source-name]'
source:
type: source_name
config:
# Required fields
option1: value1
sink:
# sink configs
View All Recipe Configuartion Options
| Field | Required | Default | Description | | --- | :-: | :-: | --- | | `field1` | ✅ | `default_value` | A required field with a default value | | `field2` | ❌ | `default_value` | An optional field with a default value | | `field3` | ❌ | | An optional field without a default value | | ... | | |
'acryl-datahub[source-usage-name]'
source:
type: source-usage-name
config:
# Required Fields
option1: value1
# Options
top_n_queries: 10
sink:
# sink configs
View All Recipe Configuartion Options
| Field | Required | Default | Description | | --- | :-: | :-: | --- | | `field1` | ✅ | `default_value` | A required field with a default value | | `field2` | ❌ | `default_value` | An optional field with a default value | | `field3` | ❌ | | An optional field without a default value | | ... | | |
Troubleshooting
[Common Issue]
[Provide description of common issues with this integration and steps to resolve]
Is this page helpful?