Standardize metadata on-the-fly#
This use cases runs on a LaminDB instance with populated CellType and Pathway registries. Make sure you run the GO Ontology notebook before executing this use case.
Here, we demonstrate how to standardize the metadata on-the-fly during cell type annotation and pathway enrichment analysis using these two registries.
For more information, see:
!lamin load use-cases-registries
💡 connected lamindb: testuser1/use-cases-registries
import lamindb as ln
import bionty as bt
from lamin_usecases import datasets as ds
import scanpy as sc
import matplotlib.pyplot as plt
import celltypist
import gseapy as gp
💡 connected lamindb: testuser1/use-cases-registries
sc.settings.set_figure_params(dpi=50, facecolor="white")
ln.settings.transform.stem_uid = "hsPU1OENv0LS"
ln.settings.transform.version = "0"
ln.track()
💡 notebook imports: bionty==0.42.9 celltypist==1.6.2 gseapy==1.1.2 lamin_usecases==0.0.1 lamindb==0.70.3 matplotlib==3.8.4 scanpy==1.10.1
💡 saved: Transform(uid='hsPU1OENv0LS6K79', name='Standardize metadata on-the-fly', key='analysis-registries', version='0', type='notebook', updated_at=2024-04-22 10:23:42 UTC, created_by_id=1)
💡 saved: Run(uid='UOeBpinlupgJKu2DaB34', transform_id=1, created_by_id=1)
An interferon-beta treated dataset#
A small peripheral blood mononuclear cell dataset that is split into control and stimulated groups. The stimulated group was treated with interferon beta.
Let’s load the dataset and perform some preprocessing:
adata = ds.anndata_seurat_ifnb(preprocess=False, populate_registries=True)
adata
AnnData object with n_obs × n_vars = 13999 × 9935
obs: 'stim'
var: 'symbol'
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)
sc.pp.highly_variable_genes(adata, n_top_genes=2000)
sc.pp.pca(adata, n_comps=20)
sc.pp.neighbors(adata, n_pcs=10)
sc.tl.umap(adata)
Analysis: cell type annotation using CellTypist#
model = celltypist.models.Model.load(model="Immune_All_Low.pkl")
Show code cell output
🔎 No available models. Downloading...
📜 Retrieving model list from server https://celltypist.cog.sanger.ac.uk/models/models.json
📚 Total models in list: 46
📂 Storing models in /home/runner/.celltypist/data/models
💾 Downloading model [1/46]: Immune_All_Low.pkl
💾 Downloading model [2/46]: Immune_All_High.pkl
💾 Downloading model [3/46]: Adult_CynomolgusMacaque_Hippocampus.pkl
💾 Downloading model [4/46]: Adult_Human_PancreaticIslet.pkl
💾 Downloading model [5/46]: Adult_Human_Skin.pkl
💾 Downloading model [6/46]: Adult_Mouse_Gut.pkl
💾 Downloading model [7/46]: Adult_Mouse_OlfactoryBulb.pkl
💾 Downloading model [8/46]: Adult_Pig_Hippocampus.pkl
💾 Downloading model [9/46]: Adult_RhesusMacaque_Hippocampus.pkl
💾 Downloading model [10/46]: Autopsy_COVID19_Lung.pkl
💾 Downloading model [11/46]: COVID19_HumanChallenge_Blood.pkl
💾 Downloading model [12/46]: COVID19_Immune_Landscape.pkl
💾 Downloading model [13/46]: Cells_Adult_Breast.pkl
💾 Downloading model [14/46]: Cells_Fetal_Lung.pkl
💾 Downloading model [15/46]: Cells_Human_Tonsil.pkl
💾 Downloading model [16/46]: Cells_Intestinal_Tract.pkl
💾 Downloading model [17/46]: Cells_Lung_Airway.pkl
💾 Downloading model [18/46]: Developing_Human_Brain.pkl
💾 Downloading model [19/46]: Developing_Human_Gonads.pkl
💾 Downloading model [20/46]: Developing_Human_Hippocampus.pkl
💾 Downloading model [21/46]: Developing_Human_Organs.pkl
💾 Downloading model [22/46]: Developing_Human_Thymus.pkl
💾 Downloading model [23/46]: Developing_Mouse_Brain.pkl
💾 Downloading model [24/46]: Developing_Mouse_Hippocampus.pkl
💾 Downloading model [25/46]: Fetal_Human_AdrenalGlands.pkl
💾 Downloading model [26/46]: Fetal_Human_Pancreas.pkl
💾 Downloading model [27/46]: Fetal_Human_Pituitary.pkl
💾 Downloading model [28/46]: Fetal_Human_Retina.pkl
💾 Downloading model [29/46]: Fetal_Human_Skin.pkl
💾 Downloading model [30/46]: Healthy_Adult_Heart.pkl
💾 Downloading model [31/46]: Healthy_COVID19_PBMC.pkl
💾 Downloading model [32/46]: Healthy_Human_Liver.pkl
💾 Downloading model [33/46]: Healthy_Mouse_Liver.pkl
💾 Downloading model [34/46]: Human_AdultAged_Hippocampus.pkl
💾 Downloading model [35/46]: Human_Developmental_Retina.pkl
💾 Downloading model [36/46]: Human_Embryonic_YolkSac.pkl
💾 Downloading model [37/46]: Human_IPF_Lung.pkl
💾 Downloading model [38/46]: Human_Longitudinal_Hippocampus.pkl
💾 Downloading model [39/46]: Human_Lung_Atlas.pkl
💾 Downloading model [40/46]: Human_PF_Lung.pkl
💾 Downloading model [41/46]: Lethal_COVID19_Lung.pkl
💾 Downloading model [42/46]: Mouse_Dentate_Gyrus.pkl
💾 Downloading model [43/46]: Mouse_Isocortex_Hippocampus.pkl
💾 Downloading model [44/46]: Mouse_Postnatal_DentateGyrus.pkl
💾 Downloading model [45/46]: Nuclei_Lung_Airway.pkl
💾 Downloading model [46/46]: Pan_Fetal_Human.pkl
predictions = celltypist.annotate(
adata, model="Immune_All_Low.pkl", majority_voting=True
)
adata.obs["cell_type_celltypist"] = predictions.predicted_labels.majority_voting
🔬 Input data has 13999 cells and 9935 genes
🔗 Matching reference genes in the model
🧬 3697 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
👀 Detected a neighborhood graph in the input object, will run over-clustering on the basis of it
⛓️ Over-clustering input data with resolution set to 10
🗳️ Majority voting the predictions
✅ Majority voting done!
bt.CellType.inspect(adata.obs["cell_type_celltypist"]);
❗ received 14 unique terms, 13985 empty/duplicated terms are ignored
❗ 14 terms (100.00%) are not validated for name: Intermediate macrophages, B cells, Tem/Effector helper T cells, Tem/Trm cytotoxic T cells, Non-classical monocytes, Regulatory T cells, Tcm/Naive helper T cells, Tcm/Naive cytotoxic T cells, CD16+ NK cells, pDC, DC2, Type 1 helper T cells, Classical monocytes, DC
detected 2 CellType terms in Bionty as synonyms: 'pDC', 'DC2'
→ add records from Bionty to your CellType registry via .from_values()
couldn't validate 14 terms: 'Classical monocytes', 'Regulatory T cells', 'Type 1 helper T cells', 'Tem/Trm cytotoxic T cells', 'Non-classical monocytes', 'Tcm/Naive helper T cells', 'CD16+ NK cells', 'Intermediate macrophages', 'DC2', 'DC', 'pDC', 'B cells', 'Tcm/Naive cytotoxic T cells', 'Tem/Effector helper T cells'
→ if you are sure, create new records via ln.CellType() and save to your registry
adata.obs["cell_type_celltypist"] = bt.CellType.standardize(
adata.obs["cell_type_celltypist"]
)
❗ found 2 synonyms in Bionty: ['pDC', 'DC2']
please add corresponding CellType records via `.from_values(['plasmacytoid dendritic cell'])`
# Register cell type of found synonym
bt.CellType.from_public(name='plasmacytoid dendritic cell').save()
❗ now recursing through parents: this only happens once, but is much slower than bulk saving
sc.pl.umap(
adata,
color=["cell_type_celltypist", "stim"],
frameon=False,
legend_fontsize=10,
wspace=0.4,
)
... storing 'cell_type_celltypist' as categorical
Analysis: Pathway enrichment analysis using Enrichr#
This analysis is based on the GSEApy scRNA-seq Example.
First, we compute differentially expressed genes using a Wilcoxon test between stimulated and control cells.
# compute differentially expressed genes
sc.tl.rank_genes_groups(
adata,
groupby="stim",
use_raw=False,
method="wilcoxon",
groups=["STIM"],
reference="CTRL",
)
rank_genes_groups_df = sc.get.rank_genes_groups_df(adata, "STIM")
rank_genes_groups_df.head()
| names | scores | logfoldchanges | pvals | pvals_adj | |
|---|---|---|---|---|---|
| 0 | ISG15 | 99.455093 | 7.132659 | 0.0 | 0.0 |
| 1 | ISG20 | 96.735359 | 5.074261 | 0.0 | 0.0 |
| 2 | IFI6 | 94.970978 | 5.828640 | 0.0 | 0.0 |
| 3 | IFIT3 | 92.481857 | 7.432349 | 0.0 | 0.0 |
| 4 | IFIT1 | 90.698700 | 8.053571 | 0.0 | 0.0 |
Next, we filter out up/down-regulated differentially expressed gene sets:
degs_up = rank_genes_groups_df[
(rank_genes_groups_df["logfoldchanges"] > 0)
& (rank_genes_groups_df["pvals_adj"] < 0.05)
]
degs_dw = rank_genes_groups_df[
(rank_genes_groups_df["logfoldchanges"] < 0)
& (rank_genes_groups_df["pvals_adj"] < 0.05)
]
degs_up.shape, degs_dw.shape
((541, 5), (935, 5))
Run pathway enrichment analysis on DEGs and plot top 10 pathways:
enr_up = gp.enrichr(degs_up.names, gene_sets="GO_Biological_Process_2023").res2d
gp.dotplot(enr_up, figsize=(2, 3), title="Up", cmap=plt.cm.autumn_r);
enr_dw = gp.enrichr(degs_dw.names, gene_sets="GO_Biological_Process_2023").res2d
gp.dotplot(enr_dw, figsize=(2, 3), title="Down", cmap=plt.cm.winter_r);
Register analyzed dataset and annotate with metadata#
Register new features and labels (check out more details here):
new_features = ln.Feature.from_df(adata.obs)
ln.save(new_features)
new_labels = [ln.ULabel(name=i) for i in adata.obs["stim"].unique()]
ln.save(new_labels)
features = ln.Feature.lookup()
Register dataset using a Artifact object:
artifact = ln.Artifact.from_anndata(
adata,
description="seurat_ifnb_activated_Bcells",
)
artifact.save()
artifact.features.add_from_anndata(
var_field=bt.Gene.symbol,
organism="human", # optionally, globally set organism via bt.settings.organism = "human"
)
Link cell type labels#
cell_type_records = bt.CellType.from_values(adata.obs["cell_type_celltypist"])
artifact.labels.add(cell_type_records, features.cell_type_celltypist)
❗ did not create CellType records for 12 non-validated names: 'Classical monocytes', 'Regulatory T cells', 'Tcm/Naive cytotoxic T cells', 'Type 1 helper T cells', 'Tem/Trm cytotoxic T cells', 'Non-classical monocytes', 'Tcm/Naive helper T cells', 'Intermediate macrophages', 'DC', 'B cells', 'CD16+ NK cells', 'Tem/Effector helper T cells'
Link stimulation labels:
stim_records = ln.ULabel.from_values(adata.obs["stim"])
artifact.labels.add(stim_records, features.stim)
Link pathway labels#
Let’s enable tracking of the current notebook as the transform of this artifact:
We further create two feature sets for degs_up and degs_dw which we can later associate with the associated pathways:
degs_up_featureset = ln.FeatureSet.from_values(
degs_up.names,
bt.Gene.symbol,
name="Up-regulated DEGs STIM vs CTRL",
type="category",
organism=( # optionally, globally set organism via bt.settings.organism = "human"
"human"
),
)
degs_dw_featureset = ln.FeatureSet.from_values(
degs_dw.names,
bt.Gene.symbol,
name="Down-regulated DEGs STIM vs CTRL",
type="category",
organism=( # optionally, globally set organism via bt.settings.organism = "human"
"human"
),
)
# Link feature sets to artifact
artifact.features._add_feature_set(degs_up_featureset, slot="STIM-up-DEGs")
artifact.features._add_feature_set(degs_dw_featureset, slot="STIM-down-DEGs")
Link the top 10 pathways to the corresponding differentially expressed genes:
def parse_ontology_id_from_enrichr_results(key):
"""Parse out the ontology id.
"ATF6-mediated Unfolded Protein Response (GO:0036500)" -> ("GO:0036500", "ATF6-mediated Unfolded Protein Response")
"""
id = key.split(" ")[-1].replace("(", "").replace(")", "")
name = key.replace(f" ({id})", "")
return (id, name)
# get ontology ids for the top 10 pathways
enr_up_top10 = [
pw_id[0]
for pw_id in enr_up.head(10).Term.apply(parse_ontology_id_from_enrichr_results)
]
enr_dw_top10 = [
pw_id[0]
for pw_id in enr_dw.head(10).Term.apply(parse_ontology_id_from_enrichr_results)
]
# get pathway records
enr_up_top10_pathways = bt.Pathway.from_values(enr_up_top10, bt.Pathway.ontology_id)
enr_dw_top10_pathways = bt.Pathway.from_values(enr_dw_top10, bt.Pathway.ontology_id)
Associate the pathways to the differentially expressed genes:
degs_up_featureset.pathways.set(enr_up_top10_pathways)
degs_dw_featureset.pathways.set(enr_dw_top10_pathways)
degs_up_featureset.pathways.list("name")
['defense response to symbiont',
'response to type II interferon',
'defense response to virus',
'negative regulation of viral genome replication',
'cellular response to cytokine stimulus',
'positive regulation of cytokine production',
'response to interferon-beta',
'negative regulation of viral process',
'response to cytokine',
'regulation of viral genome replication']
Querying metadata#
artifact.describe()
Artifact(uid='leuBAOMuIcxYWhnPO8KD', suffix='.h5ad', accessor='AnnData', description='seurat_ifnb_activated_Bcells', size=214889953, hash='BipY8IBYLjV_RjUkPLYFYo', hash_type='sha1-fl', visibility=1, key_is_virtual=True, updated_at=2024-04-22 10:26:39 UTC)
Provenance:
📎 storage: Storage(uid='ydOXaWOT', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/use-cases-registries', type='local')
📎 transform: Transform(uid='hsPU1OENv0LS6K79', name='Standardize metadata on-the-fly', key='analysis-registries', version='0', type='notebook')
📎 run: Run(uid='UOeBpinlupgJKu2DaB34', started_at=2024-04-22 10:23:42 UTC, is_consecutive=True)
📎 created_by: User(uid='DzTjkKse', handle='testuser1', name='Test User1')
Features:
var: FeatureSet(uid='YUACiawIbahhw9mP8GMX', n=11276, type='number', registry='bionty.Gene')
'GFI1', 'ICMT', 'IFNGR2', 'CADM1', 'PHF19', 'CCNY', 'CDO1', 'APOA1', 'IFNB1', 'SPINK2', 'USP10', 'PRELID2', 'RPUSD4', 'ARHGAP25', 'MED9', 'PKN1', 'S100B', 'GID4', 'JUN', 'LIN9', ...
obs: FeatureSet(uid='nL8FY9k49XtJPCvsddS6', n=2, registry='core.Feature')
🔗 stim (2, core.ULabel): 'CTRL', 'STIM'
🔗 cell_type_celltypist (1, bionty.CellType): 'plasmacytoid dendritic cell'
STIM-up-DEGs: FeatureSet(uid='60S7Nj9oWlNmagiSGs0Q', name='Up-regulated DEGs STIM vs CTRL', n=660, type='category', registry='bionty.Gene')
'NT5C3A', 'IFI16', 'RGL1', 'ODF2L', 'KLF6', 'HPSE', 'TIA1', 'SAP18', 'GBP5', 'SH3GLB1', 'PSMA5', 'SYNE2', 'ARHGAP25', 'CH25H', 'ANXA4', 'MYC', 'GNAI3', 'HBP1', 'SCO2', 'RPL28', ...
STIM-down-DEGs: FeatureSet(uid='dWRmT1QmoqyQXg9y7Ght', name='Down-regulated DEGs STIM vs CTRL', n=1093, type='category', registry='bionty.Gene')
'COPS6', 'DAZAP2', 'ASGR1', 'RPS3A', 'PIN1', 'SLC43A2', 'IFNGR2', 'RPL7', 'MLF2', 'GNAI2', 'UQCRC1', 'HLA-DRA', 'NDUFS8', 'FUCA1', 'MFSD10', 'PGLS', 'SLC25A11', 'PSMA7', 'RPS6', 'DEDD2', ...
Labels:
📎 cell_types (1, bionty.CellType): 'plasmacytoid dendritic cell'
📎 ulabels (2, core.ULabel): 'CTRL', 'STIM'
Querying cell types#
Querying for cell types contains “B cell” in the name:
bt.CellType.filter(name__contains="B cell").df().head()
| uid | name | ontology_id | abbr | synonyms | description | created_at | updated_at | public_source_id | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|
| id |
Querying for all artifacts annotated with a cell type:
celltypes = bt.CellType.lookup()
celltypes.plasmacytoid_dendritic_cell
CellType(uid='3JO0EdVd', name='plasmacytoid dendritic cell', ontology_id='CL:0000784', synonyms='type 2 DC|pDC|interferon-producing cell|IPC|T-associated plasma cell|plasmacytoid T cell|DC2|plasmacytoid monocyte|lymphoid dendritic cell', description='A Dendritic Cell Type Of Distinct Morphology, Localization, And Surface Marker Expression (Cd123-Positive) From Other Dendritic Cell Types And Associated With Early Stage Immune Responses, Particularly The Release Of Physiologically Abundant Amounts Of Type I Interferons In Response To Infection.', updated_at=2024-04-22 10:26:20 UTC, public_source_id=21, created_by_id=1)
ln.Artifact.filter(cell_types=celltypes.plasmacytoid_dendritic_cell).df()
| uid | storage_id | key | suffix | accessor | description | version | size | hash | hash_type | n_objects | n_observations | transform_id | run_id | visibility | key_is_virtual | created_at | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||||||||||||
| 1 | leuBAOMuIcxYWhnPO8KD | 1 | None | .h5ad | AnnData | seurat_ifnb_activated_Bcells | None | 214889953 | BipY8IBYLjV_RjUkPLYFYo | sha1-fl | None | None | 1 | 1 | 1 | True | 2024-04-22 10:26:37.616712+00:00 | 2024-04-22 10:26:39.261602+00:00 | 1 |
Querying pathways#
Querying for pathways contains “interferon-beta” in the name:
bt.Pathway.filter(name__contains="interferon-beta").df()
| uid | name | ontology_id | abbr | synonyms | description | public_source_id | created_at | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||
| 684 | 1l4z0v8W | cellular response to interferon-beta | GO:0035458 | None | cellular response to fibroblast interferon|cel... | Any Process That Results In A Change In State ... | 48 | 2024-04-22 10:23:03.694426+00:00 | 2024-04-22 10:23:03.694435+00:00 | 1 |
| 2130 | 1NzHDJDi | negative regulation of interferon-beta production | GO:0032688 | None | down regulation of interferon-beta production|... | Any Process That Stops, Prevents, Or Reduces T... | 48 | 2024-04-22 10:23:03.985043+00:00 | 2024-04-22 10:23:03.985053+00:00 | 1 |
| 3127 | 3x0xmK1y | positive regulation of interferon-beta production | GO:0032728 | None | positive regulation of IFN-beta production|up-... | Any Process That Activates Or Increases The Fr... | 48 | 2024-04-22 10:23:04.098676+00:00 | 2024-04-22 10:23:04.098685+00:00 | 1 |
| 4334 | 54R2a0el | regulation of interferon-beta production | GO:0032648 | None | regulation of IFN-beta production | Any Process That Modulates The Frequency, Rate... | 48 | 2024-04-22 10:23:04.229955+00:00 | 2024-04-22 10:23:04.229964+00:00 | 1 |
| 4953 | 3VZq4dMe | response to interferon-beta | GO:0035456 | None | response to fiblaferon|response to fibroblast ... | Any Process That Results In A Change In State ... | 48 | 2024-04-22 10:23:04.299747+00:00 | 2024-04-22 10:23:04.299771+00:00 | 1 |
Query pathways from a gene:
bt.Pathway.filter(genes__symbol="KIR2DL1").df()
| uid | name | ontology_id | abbr | synonyms | description | public_source_id | created_at | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||
| 1346 | 7S7qlEkG | immune response-inhibiting cell surface recept... | GO:0002767 | None | immune response-inhibiting cell surface recept... | The Series Of Molecular Signals Initiated By A... | 48 | 2024-04-22 10:23:03.764748+00:00 | 2024-04-22 10:23:03.764757+00:00 | 1 |
Query artifacts from a pathway:
ln.Artifact.filter(feature_sets__pathways__name__icontains="interferon-beta").first()
Artifact(uid='leuBAOMuIcxYWhnPO8KD', suffix='.h5ad', accessor='AnnData', description='seurat_ifnb_activated_Bcells', size=214889953, hash='BipY8IBYLjV_RjUkPLYFYo', hash_type='sha1-fl', visibility=1, key_is_virtual=True, updated_at=2024-04-22 10:26:39 UTC, storage_id=1, transform_id=1, run_id=1, created_by_id=1)
Query featuresets from a pathway to learn from which geneset this pathway was computed:
pathway = bt.Pathway.filter(ontology_id="GO:0035456").one()
pathway
Pathway(uid='3VZq4dMe', name='response to interferon-beta', ontology_id='GO:0035456', synonyms='response to fiblaferon|response to fibroblast interferon|response to interferon beta', description='Any Process That Results In A Change In State Or Activity Of A Cell Or An Organism (In Terms Of Movement, Secretion, Enzyme Production, Gene Expression, Etc.) As A Result Of An Interferon-Beta Stimulus. Interferon-Beta Is A Type I Interferon.', updated_at=2024-04-22 10:23:04 UTC, public_source_id=48, created_by_id=1)
degs = ln.FeatureSet.filter(pathways__ontology_id=pathway.ontology_id).one()
Now we can get the list of genes that are differentially expressed and belong to this pathway:
contributing_genes = pathway.genes.all() & degs.genes.all()
contributing_genes.list("symbol")
['PLSCR1',
'IFITM3',
'MNDA',
'XAF1',
'IFITM2',
'IFITM1',
'STAT1',
'CALM1',
'OAS1',
'BST2',
'IRF1',
'PNPT1',
'AIM2',
'IFI16',
'SHFL']
# clean up test instance
!lamin delete --force use-cases-registries
!rm -r ./use-cases-registries
Show code cell output
💡 deleting instance testuser1/use-cases-registries
❗ manually delete your stored data: /home/runner/work/lamin-usecases/lamin-usecases/docs/use-cases-registries