MongoDB
Important Capabilities
Capability | Status | Notes |
---|---|---|
Schema Metadata | ✅ | Enabled by default |
This plugin extracts the following:
- Databases and associated metadata
- Collections in each database and schemas for each collection (via schema inference)
By default, schema inference samples 1,000 documents from each collection. Setting schemaSamplingSize: null
will scan the entire collection.
Moreover, setting useRandomSampling: False
will sample the first documents found without random selection, which may be faster for large collections.
Note that schemaSamplingSize
has no effect if enableSchemaInference: False
is set.
Really large schemas will be further truncated to a maximum of 300 schema fields. This is configurable using the maxSchemaSize
parameter.
CLI based Ingestion
Install the Plugin
pip install 'acryl-datahub[mongodb]'
Starter Recipe
Check out the following recipe to get started with ingestion! See below for full configuration options.
For general pointers on writing and running a recipe, see our main recipe guide.
source:
type: "mongodb"
config:
# Coordinates
connect_uri: "mongodb://localhost"
# Credentials
username: admin
password: password
authMechanism: "DEFAULT"
# Options
enableSchemaInference: True
useRandomSampling: True
maxSchemaSize: 300
sink:
# sink configs
Config Details
- Options
- Schema
Note that a .
is used to denote nested fields in the YAML recipe.
Field | Description |
---|---|
authMechanism string | MongoDB authentication mechanism. |
connect_uri string | MongoDB connection URI. Default: mongodb://localhost |
enableSchemaInference boolean | Whether to infer schemas. Default: True |
maxDocumentSize integer | Default: 16793600 |
maxSchemaSize integer | Maximum number of fields to include in the schema. Default: 300 |
options object | Additional options to pass to pymongo.MongoClient() . Default: {} |
password string | MongoDB password. |
schemaSamplingSize integer | Number of documents to use when inferring schema size. If set to 0 , all documents will be scanned. Default: 1000 |
useRandomSampling boolean | If documents for schema inference should be randomly selected. If False , documents will be selected from start. Default: True |
username string | MongoDB username. |
env string | The environment that all assets produced by this connector belong to Default: PROD |
collection_pattern AllowDenyPattern | regex patterns for collections to filter in ingestion. Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True} |
collection_pattern.allow array(string) | |
collection_pattern.deny array(string) | |
collection_pattern.ignoreCase boolean | Whether to ignore case sensitivity during pattern matching. Default: True |
database_pattern AllowDenyPattern | regex patterns for databases to filter in ingestion. Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True} |
database_pattern.allow array(string) | |
database_pattern.deny array(string) | |
database_pattern.ignoreCase boolean | Whether to ignore case sensitivity during pattern matching. Default: True |
The JSONSchema for this configuration is inlined below.
{
"title": "MongoDBConfig",
"description": "Any source that produces dataset urns in a single environment should inherit this class",
"type": "object",
"properties": {
"env": {
"title": "Env",
"description": "The environment that all assets produced by this connector belong to",
"default": "PROD",
"type": "string"
},
"connect_uri": {
"title": "Connect Uri",
"description": "MongoDB connection URI.",
"default": "mongodb://localhost",
"type": "string"
},
"username": {
"title": "Username",
"description": "MongoDB username.",
"type": "string"
},
"password": {
"title": "Password",
"description": "MongoDB password.",
"type": "string"
},
"authMechanism": {
"title": "Authmechanism",
"description": "MongoDB authentication mechanism.",
"type": "string"
},
"options": {
"title": "Options",
"description": "Additional options to pass to `pymongo.MongoClient()`.",
"default": {},
"type": "object"
},
"enableSchemaInference": {
"title": "Enableschemainference",
"description": "Whether to infer schemas. ",
"default": true,
"type": "boolean"
},
"schemaSamplingSize": {
"title": "Schemasamplingsize",
"description": "Number of documents to use when inferring schema size. If set to `0`, all documents will be scanned.",
"default": 1000,
"exclusiveMinimum": 0,
"type": "integer"
},
"useRandomSampling": {
"title": "Userandomsampling",
"description": "If documents for schema inference should be randomly selected. If `False`, documents will be selected from start.",
"default": true,
"type": "boolean"
},
"maxSchemaSize": {
"title": "Maxschemasize",
"description": "Maximum number of fields to include in the schema.",
"default": 300,
"exclusiveMinimum": 0,
"type": "integer"
},
"maxDocumentSize": {
"title": "Maxdocumentsize",
"default": 16793600,
"exclusiveMinimum": 0,
"type": "integer"
},
"database_pattern": {
"title": "Database Pattern",
"description": "regex patterns for databases to filter in ingestion.",
"default": {
"allow": [
".*"
],
"deny": [],
"ignoreCase": true
},
"allOf": [
{
"$ref": "#/definitions/AllowDenyPattern"
}
]
},
"collection_pattern": {
"title": "Collection Pattern",
"description": "regex patterns for collections to filter in ingestion.",
"default": {
"allow": [
".*"
],
"deny": [],
"ignoreCase": true
},
"allOf": [
{
"$ref": "#/definitions/AllowDenyPattern"
}
]
}
},
"additionalProperties": false,
"definitions": {
"AllowDenyPattern": {
"title": "AllowDenyPattern",
"description": "A class to store allow deny regexes",
"type": "object",
"properties": {
"allow": {
"title": "Allow",
"description": "List of regex patterns to include in ingestion",
"default": [
".*"
],
"type": "array",
"items": {
"type": "string"
}
},
"deny": {
"title": "Deny",
"description": "List of regex patterns to exclude from ingestion.",
"default": [],
"type": "array",
"items": {
"type": "string"
}
},
"ignoreCase": {
"title": "Ignorecase",
"description": "Whether to ignore case sensitivity during pattern matching.",
"default": true,
"type": "boolean"
}
},
"additionalProperties": false
}
}
}
Code Coordinates
- Class Name:
datahub.ingestion.source.mongodb.MongoDBSource
- Browse on GitHub
Questions
If you've got any questions on configuring ingestion for MongoDB, feel free to ping us on our Slack.