## Changelog 2025 Note: Get notified by watching releases for git repositories: lamindb · laminhub · lamin-cli · laminr · nf-lamin · bionty For other years, see: 2026 · 2024 · 2023 · 2022 # 2025-12-18 hub 1.6.0 * ✨ Creating sheets from scratch + creating schemas & features + updating features of a sheet PR @falexwolf * 🐛 Fix error on instance "Settings" page if no schema modules are defined PR @chaichontat * 💄 Update style for "New instance" button PR @chaichontat # 2025-12-16 hub 1.5.2 * 🐛 Fix ancestry resolution for nested "Record[...]" dtypes PR @chaichontat * 🚸 Always show delete confirmation prompt in sheets PR @chaichontat # 2025-12-15 R 1.2.1 * 🐛 Adapt to change of "SQLRecord.describe()" method in "lamindb" 1.17.0 PR @lazappi * 🚸 Fix expected version check PR @lazappi # 2025-12-15 hub 1.5.1 * 🐛 Enforce branch code filter also when "showAll=true" PR @chaichontat * 🐛 Access tables now show a loading state PR @chaichontat * ⚡️ Faster load for instance collaborators PR @chaichontat # 2025-12-14 db 1.17.0 -[ ⚠️ Run "lamin migrate deploy" ]- All instances connected to LaminHub have been migrated and there is no need to act. If you are an admin of a self-managed instance, please migrate your database with "lamin migrate deploy". For details see the Compatibility Matrix and the source code. LaminDB instances can now easily be snapshot as an SQLite file. We leverage this to provide "https" access to clones of public instances that are hosted on S3, avoiding firewall-related time outs if the Postgres-protocol or ports are blacklisted and generally improving latency of queries. * ✨ Add "ln.DB" to conveniently query all entities in any instance PR @Zethson * ✨ Add "lamin io" to manage snapshots, exports & imports PR @Zethson * ✨ Read-only queries of public instances through shadow SQLite clones PR @Zethson More convenient access to feature values: * ✨ Implement "FeatureManager.__getitem__" for getting values records PR @sunnyosun General UX improvements: * 🚸 Enable queries to other instances also in case no default instance is configured PR @Koncopd * 🚸 Introduce ".describe()" on all "BaseSQLRecord" objects and classes + on "QuerySet" PR @falexwolf * 🚸 Warn if a user passes a string to the "dtype" argument of "Feature" and add validation for existence of record types PR @falexwolf * 🚸 No more warning about not being connected given "ln.DB()" now exists PR @falexwolf General data integrity: * 🚸 Propagate re-organization of record types to dtypes PR @falexwolf * 🚸 Maintain a history of feature renaming events in the feature record, simplify handling label rename events PR @falexwolf * 🚸 When deleting a record type, delete its descendants, too PR @falexwolf * 🚸 Be more strict about constraining types in "HasType" PR @falexwolf Validation & annotation: * 🚸 Make adding optional features convenient through "schema.add_optional_features()" PR @falexwolf * 🚸 Better handle "organism" constraints where they are needed, default to "bt.settings.organism = None" PR @sunnyosun * 🚸 Improve error behavior if a "type" isn't found during validation: raise an "IntegrityError" instead of "InvalidArgument" PR @falexwolf * 🚸 Leverage schema feature association when annotating artifacts through curators PR @falexwolf * 🚸 If matching the same term multiple times at different levels of a type hierarchy during validation, annotate with top-level match PR @falexwolf * 🐛 Resolve feature name clashes during usage of "artifact.features.add_values()" PR @falexwolf * 🐛 Fix a bug where a changed set of valid features led to violating the per-"slot" unique constraint in "ArtifactSchema" rather than updating the existing inferred schema link PR @falexwolf Sheets: * 🚸 Better name for "record.type_to_dataframe()": "record.to_dataframe()" PR @falexwolf * 🗃️ Add an "extra_data" JSON field to "Record" PR @falexwolf * 🐛 Gracefully handle corrupted "date" and "datetime" values in sheets PR @falexwolf * 🐛 Fix type casting to pandas in sheet exports with "dtype="num"" PR @falexwolf * 🐛 Consistent filtering behavior for "kind__startswith='__lamindb_'" artifacts across "to_dataframe()" in different contexts PR @falexwolf * 🐛 Respect soft delete in sheet export PR @falexwolf * 🐛 Fix sheet export with mixed single- and multi-item lists PR @falexwolf * 🐛 Fix sheet export when multiple features with the same name in an instance PR @falexwolf * 🐛 Fix sheet export of columns that contain only nulls and add a broad test suite PR @falexwolf * 🐛 Honor "list" "dtype" over the default "set" in sheet export PR @falexwolf * 🐛 Fix sheet export of features with list dtype and additional field qualifiers PR @falexwolf * 🐛 Fix a bug where when a list dtype categorical feature was present in a sheet without the corresponding non-list one, it'd be exlcuded from the sheet export PR @falexwolf * 🐛 In sheet export, do not limit to just 100 records PR @falexwolf * ♻️ Allow "__lamindb_..." type columns in sheet export also if "schema.maximal_set = True" PR @falexwolf General UX: * ⚡️ Faster "lamin init" from compressed SQLite file PR @ap-- @falexwolf * 🚸 Suggest using ".from_dataframe()" when passing a "DataFrame"-like path to "Artifact()" PR @falexwolf * 🚸 Give a hint on re-login while connected to an instance PR @Koncopd * 🚸 Allow omitting "entity" in "lamin load" PR @falexwolf * 🚸 Enable "skip_hash_lookup" in transform PR @falexwolf * 🚸 Show objects when storage is non-empty on deletion PR @Koncopd * 🚸 Clarify DB vendor vs. dialect PR @falexwolf * 🚸 Better error message if notebook filename is "Untitled.ipynb" PR @falexwolf * 🚸 Support ".tar.gz" PR @Koncopd * 🚸 Allow permanent deletion from CLI PR @falexwolf General bug fixes: * 🐛 Fix "logger.mute()" before importing lamindb and refactor "RootLogger" PR @falexwolf * 🐛 "Artifact()" incorrectly returned latest version instead of the matching one when "hash" was found in version history and "key" matched both versions PR @falexwolf * 🐛 Honor "dev-dir" in "lamin load" PR @falexwolf * 🐛 Honor "dev-dir" in "lamin save" PR @falexwolf * 🐛 Fix "view_parents()" and "query_relatives()" for non-default instances PR @Zethson Migrations: * 🗃️ Add multiple fields to "Transform" PR @falexwolf * 🗃️ Add "ArtifactArtifact" and "ArtifactRun" to enable "dtype=Artifact" and "dtype=Run" in artifact relationships PR @falexwolf * 🗃️ Add "RecordCollection" and "CollectionRecord" PR @falexwolf General docs: * 📝 More extensive documentation for "Artifact.otype" PR @falexwolf General refactoring: * ♻️ Allow skipping to mark the storage root when registering a storage PR @Koncopd * ♻️ Add "query_ulabels()", "query_features()", "query_schemas()", etc. to all "HasType" registries PR @falexwolf * ♻️ Add "LAMIN_S3_ANON" env variable to force "anon" mode for all non-managed s3 buckets PR @Koncopd * ♻️ Centralize "Curator" unit tests in the "core" worker PR @falexwolf * ♻️ Adapt code to rate limiting initialization of instances PR @Koncopd * ♻️ Remove logic that checks admin permissions client side before migrate step PR @falexwolf * ♻️ Corrections and clarifications for "get_parents()" PR @Koncopd * ♻️ Harmonize lineage graph display PR @falexwolf Dependencies: * ➕ Add "pgtrigger" PR @falexwolf * ⬇️ Upper bound for pandas 3 PR @Zethson Independent tests: * ✅ Also run core unit tests against an SQLite database, not just against Postgres PR @falexwolf * ✅ Test proxies and certificates with "mitmproxy" PR @Koncopd # 2025-12-13 hub 1.5.0 * ✨ Add a "Create instance" button to the UI PR @fredericenard * ✨ Introduce guests in an organization PR @fredericenard * ✨ Add per-account instance quota enforcement PR @fredericenard * ✨ Support date filling PR @chaichontat * 🚸 A specialized dict cell in sheets PR @chaichontat * 🚸 Improve the design of the account storage & infrastructure pages PR @fredericenard * 🚸 Improve robustness of sheet-to-artifact export PR @fredericenard * 🔒 Propagate org membership loss to team members and instance & space collaborators PR @fredericenard * 🐛 Enforce absolute "type" hierarchy within relational "dtypes" PR @chaichontat * 🐛 Default for "Feature.nullable" should be "True" PR @sunnyosun * 🐛 Reject empty non-nullable features with dtype "list[...]" PR @chaichontat * 🐛 Fix edge cases in the S3 storage metadata endpoint PR @chaichontat * 🐛 Skip existing transform name validation PR @chaichontat * 🐛 Bar chart renders as horizontal stacked bar with long category labels PR @sunnyosun * 💄 Fix pagination display in team members management dialog PR @sunnyosun * 💄 Refine landing page PR @falexwolf # 2025-12-04 hub 1.4.2 * 🐛 Fix a breadth of edge cases for sheet export PR @falexwolf # 2025-12-04 hub 1.4.1 * 🐛 Fix populating sheets from transforms in presence of list dtypes PR @chaichontat * 🐛 Fix sheet export for list dtypes by upgrading lamindb PR @fredericenard # 2025-12-03 hub 1.4.0 * ✨ In "Launch" view: add "list" and "dict" "dtype" support for run metadata PR @chaichontat * 🚸 In "Launch" view: better design for "select" and "create" buttons and behaviors PR @chaichontat * 🐛 Fix list feature parsing from function response PR @sunnyosun * 🐛 In CSV validation view, do not mark missing values as required for nullable and optional features PR @falexwolf # 2025-12-02 hub 1.3.1 * 🐛 Fix data propagation in "dtype" "list" fields PR @chaichontat # 2025-12-02 hub 1.3.0 * ✨ In "Sheet" editor and "csv" import, support storing extra data e.g. from non-validated columns PR @chaichontat # 2025-12-01 nf 0.2.2 * ✨ Add "getInstanceSlug()" function PR @rcannood * ✨ Allow fetching artifact paths from UID with "getArtifactFromUid" PR @hcannoodt * ✨ Implement call for fetching storage credentials PR @rcannood * 🐛 Fix some failed runs being marked as successful PR @rcannood * 🐛 Fill in "null"s in generated Transform metadata PR @rcannood * 🐛 Fix provider in build config PR @rcannood # 2025-12-01 hub 1.2.0 * ✨ Support display of list "dtype" in Records PR @chaichontat * 🚸 On "Launch" page, consolidate buttons and allow dialogs to fill page PR @chaichontat * 🚸 On "Launch" page, rename "Import to Sheet" to "Edit in browser" and add "Select" button PR @sunnyosun * 🚸 On "Launch" page, set initial artifact filter to schema name PR @chaichontat * 🐛 In lineage viewer, show deeper lineage for collections PR @falexwolf # 2025-11-27 hub 1.1.1 * 🐛 Preserve column name "__lamindb_record_name__" in csv upload PR @chaichontat # 2025-11-26 hub 1.1.0 * ✨ Directly load artifacts and files into sheets PR @chaichontat * ✨ Support executable transform UIDs PR @chaichontat * ✨ Recognize column name "__lamindb_record_name__" in csvs to indicate record names for sheet imports PR @chaichontat * 🚸 Auto-filename generation for uploads PR @chaichontat * 🚸 Improve visual distinction between optional and required features for a schema PR @chaichontat * 💄 Render images in markdown PR @sunnyosun * 💄 Render markdown bullets PR @sunnyosun # 2025-11-23 hub 1.0.0 LaminHub is a data collaboration hub built on LaminDB, similar to how GitHub is built on git. It provides a GUI for LaminDB and makes it easy to manage many LaminDB instances including hosting, access management, and soon, audit logs. Through the open-source LaminDB, developers retain full access to their data with zero lock-in danger. Instead of depending on rate- limited REST APIs, developers build directly with Postgres & diverse open storage formats on object stores. You can give it a try by exploring public omics data collections at lamin.ai/explore. It's free and no account is required. LaminHub is a SaaS product. For private data & commercial usage, see: lamin.ai/pricing. -[ Incremental changes since "0.52.0" ]- * 🚸 Make simple records clickable and add markdown page to record detailed view PR @falexwolf * 💄 Do not include "cat[" and "cat[Record[" in "dtype" display PR @falexwolf * 💄 Fix help message text color and alignment PR @sunnyosun # 2025-11-20 hub 0.52.0 * 🚸 Hide entrypoint selection if there is only a single one, rename "function" to "entrypoint" in transform launch PR @falexwolf * 🚸 Update type requirement to use an artifact type for execution PR @chaichontat * 🐛 Exclude deleted record types when determining dtypes PR @falexwolf # 2025-11-19 hub 0.51.0 * 🚸 Implement collapsible sections for "Launch" page PR @chaichontat * 🐛 Fix editing context in quick mode on "Launch" page PR @chaichontat # 2025-11-18 hub 0.50.0 * 🚸 Select artifacts in transform execute PR @chaichontat * 🚸 Better annotation definition in the "Schema" page PR @chaichontat # 2025-11-17 hub 0.49.0 * 🚸 Consolidate create new/select existing sheet logic PR @chaichontat * 🚸 Inline form for run metadata PR @chaichontat * 🚸 Auto-append run "uid" to Nextflow launch output dir PR @chaichontat * 🐛 Add autoscroll to fill handler PR @chaichontat * 🐛 Ensure a new run is visible right after launching it PR @chaichontat * 🐛 Fix faceted branch filtering PR @chaichontat * 🐛 Prevent a selector from creating new entities through PR @chaichontat * 🐛 Fix direct links to individual ulabels PR @chaichontat # 2025-11-16 hub 0.48.0 * 🚸 Create a tooltip to help orient the user in the "Create record" dialogue PR @chaichontat * 🚸 Auto-select unique transforms during sheet generation PR @chaichontat * 🐛 In lineage viewer, fix missing display of downstream lineage for collections PR @falexwolf * 🐛 Fix record labels not showing on artifact detail page PR @falexwolf * 💄 Enhance color for "Record" icon PR @falexwolf * 💄 Make it easy to create a single record PR @falexwolf * 💄 Consistently display records under root vs. sub types & exclude types from sheets PR @falexwolf * 💄 Add display of "date" and "datetime" values in sheets PR @falexwolf * 💄 Show Records at the top of the filtering sidebar for Artifact PR @falexwolf # 2025-11-14 db 1.16.2 * 🐛 Fix re-validating an existing artifact with an external schema and external features PR @falexwolf # 2025-11-13 db 1.16.1 Bug fixes: * 🐛 Do not log CLI args in notebooks PR @falexwolf * 🐛 Fix ignoring proxy settings from env variables PR @Koncopd * 🐛 Fix "Artifact" from "http" URL if leveraging virtual keys PR @Koncopd * 🐛 Fix no mimetype for URL in "get_stat_file_cloud" PR @Koncopd UX: * 🚸 Log if switching to read-only storage location of another instance also when using "ln.settings.storage" PR @falexwolf # 2025-11-12 db 1.16.0 Features: * ✨ Introduce "artifact.features.set_values()" PR @falexwolf * ✨ Make it easy to re-run scripts by tracking "cli_args" PR @falexwolf UX: * 🚸 Pass "string" to "from_source()" for look up across all semantic fields, no longer do that upon passing "name" or any other specific field PR @sunnyosun * 🚸 Enable passing a list of record objects to "record.features.add_values()" PR @falexwolf Bug fixes: * 🐛 More scalable "query_children()" and "query_parents()" PR @Koncopd Deprecations: * 🚚 Deprecate "Registry.using()" in favor of "Registry.connect()" PR @falexwolf * 🚚 Deprecate the "data" argument of "Artifact" in favor of "path" and polish class docstrings PR @falexwolf * 🚚 Deprecate "itype='Composite'" PR @falexwolf Dependencies: * ⬆️ No longer upper-bound pyarrow by "22.0" PR @Zethson * ⬆️ Support anndata up to 0.12.5 PR @Koncopd Refactoring: * ♻️ Refactor "artifact" tests PR @falexwolf * ♻️ Re-organize ".features.add_values()" and ".features.remove_values()" tests and make consistent across "Artifact" & "Record" PR @falexwolf * ♻️ Refactor and simplify "FeatureManager.remove_values()" PR @falexwolf # 2025-11-05 db 1.15.2 Note: "v1.15.1" had an unnecessary print statement, producing unexpected logs in "to_dataframe(include="features")". Bug fixes. * 🐛 Correctly deal with "None", "date", and "datetime" in "record.features.add_values()" PR @falexwolf * 🐛 Cast "int" and "float" when loading from database where appropriate PR @falexwolf * 🐛 Reset federated AWS credentials cache after logout or re-login PR @Koncopd * 🐛 Deal with corrupted state in which no "is_latest" record is found PR @falexwolf * 🐛 Do not restrict modules during "django_reset()" PR @falexwolf UX. * 🚸 Fix anndata version warnings PR @Zethson # 2025-10-30 db 1.15.0 Features. * ✨ Add "Record.features" to conveniently populate features of records PR1 PR2 @falexwolf UX: * 🚸 Improve the experience of working in a local development directory: "dev-dir" (called "work-dir" in beta) PR @falexwolf * 🚸 Support external features in "DataFrameCurator" PR @falexwolf * 🚸 Enable passing "parquet_kwargs" to "Artifact.from_dataframe()" for configuring "pyarrow.ParquetWriter" PR1 PR2 @falexwolf * 🚸 Implement database access debugging function PR @Koncopd Database: * 🗃️ Remove "db_index" on "Reference.text" field PR @falexwolf Bug fixes: * 🐛 Fix "to_dataframe()" with features PR @Koncopd * 🐛 Handle "run=None" correctly when populating subsequent creating runs of collections PR @falexwolf * 🐛 Allow "features.remove_values()" to work with features of "dtype=list[cat]" PR @sunnyosun Performance: * ⚡️ More optimal recursive childrens query PR @falexwolf Docs: * 📝 Simplify API reference PR @falexwolf * 📝 Refactor "Record" docs PR @falexwolf # 2025-10-17 nf 0.2.1 * ✨ Link Nextflow report to run PR @rcannood * ✨ Add dry_run setting PR @rcannood * ✨ Add "getTransformUid()" and "getRunUid()" functions PR @rcannood # 2025-10-27 db 1.14.0 Features: * ✨ Implement on-prem "create-s3" for existing instances PR @Koncopd UX: * 🚸 More consistent ".to_dataframe()" across "SQLRecord.to_dataframe()" and "QuerySet.to_datafame()", all now set a default "limit" of 100 and order by latest creation - also, one can now indicate including features via "include="features"" PR @falexwolf * 🚸 Enable users to annotate & validate without having to learn about the "Schema" concept PR @falexwolf * 🚸 Allow features to have reserved Django field names PR @sunnyosun * 🚸 Log writing in-memory objects into cache on artifact creation PR @Koncopd * 🚸 Simplify lineage & ontology graph displays PR @falexwolf * 🚸 Ask for extra storage-level delete confirmation in case "_real_key" is not "None" PR @falexwolf * 🚸 Better message upon start of a script in "ln.track()" PR @falexwolf * 🚸 Better dialogue for user delete within existing storage location PR @falexwolf * 🚸 Add fallback behavior if artifact was moved in storage PR @falexwolf * 🚸 Clearer error and logging messages for "sync_git_repo" and "ln.finish()" PR @falexwolf * 🚸 Prettier instance settings PR @falexwolf * 🚸 Enable passing "sync_git_repo" via env variable PR @falexwolf * 🚸 Return a "SQLRecordList" in "Artifact.from_dir()" PR @falexwolf * 🚸 Populate "reference" and "reference_type" for sliding transforms PR @sunnyosun Improvements to ".describe()": * 🚸 Re-design ".describe()" and refactor the related code base PR @falexwolf * 🚸 Add "transform.describe()" PR @falexwolf * 🚸 Improve "schema.describe()", harmonize with "describe()" of "Artifact", "Collection", and "Run" PR @falexwolf * 🚸 Enable "run.describe()" PR @falexwolf * 🚸 Better way of displaying the schema slot identifier type in ".describe()" PR @falexwolf * 🚸 Strip "cat" from feature "dtype" display PR @falexwolf Bug fixes: * 🐛 Attach schema when validating with external features PR @falexwolf * 🐛 Serialize "Iterable" of "SQLRecord" in params tracking PR @falexwolf * 🐛 Correctly handle empty lists in curation PR @sunnyosun * 🐛 Fix s3 upload and download problems arising from local aws configs PR @Koncopd * 🐛 Fix exporting sheets with features of dtype "User" PR @sunnyosun * 🐛 Also include "key" & "storage" location in error handling upon "skip_hash_lookup = True" PR @falexwolf * 🐛 Capture outputs of root logger in "LogStreamTracker" PR @falexwolf * 🐛 Exclude records in trash from unique constraint on "name" under a "type" and add "is_valid_type" constraints to all typed entities PR @falexwolf * 🐛 Fix a bug that prevented re-curating existing artifacts in some cases PR @falexwolf * 🐛 Set db tokens for connections in all contexts (threads, tasks), disable task-specific database wrappers in jupyter for ipykernel>=7.0.0 PR @Koncopd * 🐛 Raise error in curators when dtype is list but values are not PR @sunnyosun * 🐛 Respect "feature.dtype" in "FeatureManager.get_values()" PR @sunnyosun Dependencies: * ⬆️ Upgrade Django to 5.2 PR @falexwolf * ⬆️ Support fsspec 2025.9.0 PR @Koncopd Architecture: * 🏗️ For instances connected to the hub, they can now be migrated through the API PR @fredericenard Deprecations: * 🚚 Deprecate "dtype='cat'" in favor of "dtype=ln.Record" PR @falexwolf Configuration: * 🔧 Set "max_pool_connections=64" by default for "s3fs" filesystem PR @Koncopd Refactors: * ♻️ Allow collections with duplicated artifacts PR @falexwolf * ♻️ Move "define_valid_features.py" and similar scripts into the package PR @falexwolf * ♻️ Refactor removing features PR @falexwolf * ♻️ Add a "RecordRun" link model PR @falexwolf * ♻️ Fix type annotation and double-loading of dataframe in "DataFrameCurator" PR @falexwolf * ♻️ Enable bulk-re-creating records that would violate a unique constraint PR @sunnyosun * ♻️ Adapt curator tests to new "describe()" format PR @falexwolf * ♻️ Improve dealing with ad hoc annotations of lists of categorical features PR @falexwolf * ♻️ Add "init_local_sqlite" to create empty copies of existing Postgres instances PR @Zethson Docs: * 📝 Improve "manage-ontologies" guide PR @sunnyosun * 📝 README v4 PR @falexwolf # 2025-10-16 db 1.13.1 Features: * ✨ Add "Transform.from_git()" PR @falexwolf Bug fixes: * 🐛 Fix serializing & curating against dtypes in deeply nested type hierarchies PR @falexwolf @sunnyosun * 🐛 Never return existing transform with same "key" from trash PR @falexwolf UX: * 🚸 Add "restore()" to all registries with soft-delete PR @falexwolf * 🚸 Format on-prem UI URLs PR @falexwolf * 🚸 Better error message for trying to write on a public connection. PR @Koncopd Refactor: * ♻️ Map features based on name during transfer PR @sunnyosun Configurability: * 🔧 Introduce "skip_hash_lookup" for "Collection" PR @falexwolf * 🔧 Add "use_proxy_db" on connect in CLI PR @Koncopd * 🔧 Do not request db token when "use_root_db_user" is "True" on "connect" PR @Koncopd Dependencies: * ⬆️ Bump the lower bound on "aiobotocore" to 2.12.4 to allow "urllib" 2 PR @Koncopd # 2025-10-13 db 1.13.0 -[ ⚠️ Run "lamin migrate deploy" ]- All instances connected to LaminHub have been migrated and there is no need to act. If you are an admin of a self-managed instance, please migrate your database with "lamin migrate deploy". Necessary database migrations in this release consist in a new field. For details see the Compatibility Matrix and the source code. Features: * ✨ Enable tracking run parameters without validation through a new "Run.params" field PR @falexwolf UX: * 🚸 Allow "init()" after another instance was connected PR @falexwolf * 🚸 Limit name search upon record creation to records of same type PR @falexwolf * 🚸 Better UX for "lamin annotate" CLI command PR @falexwolf * 🚸 Ignore cycles arising between input artifacts and re-created artifacts PR @falexwolf * 🚸 Speed up "Curator.add_new_from()" PR @sunnyosun * 🚸 Retry POST requests for supabase functions PR @Koncopd * 🚸 Add clear errors for locked records, test RLS for locked records and "Block" models PR @Koncopd * 🚸 Polish logging during "ln.track()" PR @falexwolf Bug fixes: * 🐛 Require a restricted storage location when writing an "Artifact" to a restricted space PR @falexwolf * 🐛 Fix hash calculation for hybrid composite schemas, refactor external feature annotation in "Artifact.from_dataframe()" PR @falexwolf * ♻️ Allow using proxy db on "connect" PR @Koncopd * 🐛 Sanitize dataframe proxy for "dict" validation to enable schema- based validation of categoricals in dicts PR @falexwolf * 🐛 Fix annotation with features of "dtype = list[str]" PR @falexwolf * 🐛 Fix schema mutation when removing features PR @falexwolf * 🐛 Do not retrieve records from trash based on hash lookup PR @falexwolf @Koncopd Configurability: * 🔧 Introduce a "skip_hash_lookup" argument for "Artifact" PR @falexwolf * 🔧 Add an environment variable to set "instance.api_url" PR @Koncopd Dependencies: * ⬇️ Temporarily upper bound "ipykernel<7.0.0" PR @Koncopd # 2025-10-09 R 1.2.0 🚸 Simplified setup. PR @lazappi Install the "laminr" package. install.packages("laminr", dependencies = TRUE) Create a LaminDB instance: lc <- import_module("lamin_cli") lc$init(storage = "./mydata", modules = "bionty") Or if you have write access to an instance, login and connect to it: lc$login() lc$connect("/") More UX: * 🚸 Save a simplified R environment file when "ln$finish()" is called PR @lazappi Configurability: * 🔧 Add a "laminr_status()" function that returns a "laminr_status" object with information useful for debugging PR @lazappi * 🔧 Add a "get_current_lamin_settings()" function that returns the current LaminDB settings as an R list PR @lazappi * 🔧 Add "LAMINR_LAMINDB_VERSION" and "LAMINR_LAMINDB_OPTIONS" environment variables PR @lazappi Bug fixes: * 🐛 Handle missing setting values in "lamin_settings()" PR @lazappi * 🐛 Handle list columns when creating artifacts from data frames PR @lazappi * 🐛 Handle missing document context ID when detecting the current path for tracking PR @lazappi * 🐛 Make sure additional arguments are correctly passed to "require_module()" PR @lazappi * 🐛 Improve suffix matching when loading artifacts PR @lazappi Refactoring: * ♻️ The family of functions that proxied the CLI is now deprecated PR @lazappi * ♻️ Require instance modules before importing "lamindb" so they should be available PR @lazappi * ♻️ Add a "use_temporary_instance()" function PR @lazappi # 2025-10-07 db 1.12.1 * 🐛 Fix import of "lightning" integration PR @Zethson # 2025-10-07 db 1.12.0 -[ ⚠️ Run "lamin migrate deploy" ]- All instances connected to LaminHub have been migrated and there is no need to act. If you are an admin of a self-managed instance, please migrate your database with "lamin migrate deploy". Necessary database migrations in this release consist in new fields and auxiliary tables. For details see the Compatibility Matrix and the source code. Features: * ✨ Add PyTorch Lightning integration with a "Callback" class PR PR Guide @Zethson * ✨ Allow virtual keys with custom real storage keys PR @Koncopd Beta features: * ✨ Introduce a "work-dir" as a local development directory to organize scripts & notebooks PR @falexwolf Improvements to the "Record" class: * 🚸 Expand "Record" so that it can act as an ontology akin to "ULabel" PR @falexwolf * 🚸 Add conditional unique constraint for "Record.name" given "type" and "space" PR @falexwolf Database schema: * 🗃️ Enable "dtype=User" PR @falexwolf * 🗃️ Enable attaching markdown blocks to all entities PR2 @falexwolf * 🗃️ Add an "is_locked" field to "SQLRecord" PR1 PR2 @falexwolf * 🗃️ Make "description" fields unlimited length for all registries and increase the max length for "Artifact.key" and "Transform.key" to 1024 chars PR @falexwolf * 🗃️ Add a "description" field to "Project", replace "Person" registry with "Record" registry PR @falexwolf * 🗃️ Make feature "dtype" non-nullable on the database level if "is_type=False" PR @falexwolf UX: * 🚸 Add the current branch and the main branch by default for filtering PR @Koncopd * 🚸 Enable creating projects from CLI PR @falexwolf * 🚸 Consistent delete "permanent" argument across "QuerySet", "BaseSQLRecord" and "SQLRecord" PR @Koncopd * 🚸 Disable "gssencmode" for postgres PR @Koncopd * 🚸 Raise an error in "AnnDataCurator" when "schema.otype != 'AnnData'", no exceptions for "otype is None" PR @sunnyosun * 🚸 Replace default organism warning with targeted message during validation error PR @falexwolf * 🚸 Change logging for django module variable reset to debug PR @falexwolf * 🚸 Improve logging when violating versioning constraints PR @falexwolf * 🚸 De-duplicate data lineage in "Collection.cache()" PR @falexwolf Bugs: * 🐛 Fix dealing with uniqueness constraints of soft-deleted ontology records PR @sunnyosun * 🐛 Extend "query_relatives()" to respect branches PR @falexwolf * 🐛 Fix small chunksize for very large files when uploading to s3 PR @Koncopd * 🐛 Set run status code to completion upon "ln.tracked()" PR @falexwolf Code organization: * 🎨 Expose main CLI commands as functions on the "lamin_cli" root API for LaminR PR @falexwolf * 🎨 Deprecate "str_as_cat" in "Feature.from_dict()" PR @Zethson * 🎨 Do no longer allow "upload" as a transform type PR @falexwolf Dependencies: * ⬆️ Remove upper bound on "scipy" and bump "anndata" PR @Koncopd # 2025-09-30 nf 0.2.0 * ✨ Allow an advanced user to specify the transform and/or run UID in the config PR @rcannood * ✨ Update run status based on lamindb changes PR @rcannood * 🐛 Fix command detail PR @Zethson * 🐛 Push plugin results to writable bucket PR @Zethson * 🐛 Fix missing nextflow config PR @rcannood * 🐛 Remove dangling files from previous plugin mechanism PR @rcannood * 🔊 Throw more understandable error when an instance can't be found PR @rcannood * 🔈 Make logging consistent with other Lamin packages PR @falexwolf * ⬆️ Upgrade to new nextflow plugin setup PR @rcannood * 📝 Overhaul setup & documentation PR @Zethson # 2025-09-23 db 1.11.3 Note: "v1.11.2" didn't update internal dependencies correctly and was yanked on PyPI. * 🐛 Fix inconsistency between ".get()" and ".filter()" due to "branch_id" PR @Koncopd * 🐛 Add "copy()" to iteration over "sys.modules" PR @falexwolf * 🚸 Throw a clearer error when creating a schema without features & components and "is_type=True" PR @sunnyosun * 🚸 Fix the cache warning on "ln.track()" in scripts on Windows PR @Koncopd * 📝 Improve the README PR @Koncopd # 2025-09-18 db 1.11.1 * 🐛 Enable version lookup in "keep_artifacts_local" mode PR @falexwolf * 🚸 Better detection of untitled notebooks in "track()" PR @Koncopd * 🚸 Clear error message on RLS error trigger for tracking artifacts and collections PR @Koncopd # 2025-09-14 db 1.11.0 | bionty 1.7.0 | wetlab 1.5.0 🏗️ Always import the entire public API & enable re-connecting in same Python session PR1 PR2 PR3 PR4 @falexwolf @Koncopd -[ Additional package re-organization ]- Because users tend to forget commonly used extras in installation commands, a few optional dependencies are now part of the default dependencies. * 🏗️ Include "wetlab" & "bionty" in lamindb dependencies PR @falexwolf * 🏗️ Integrate the 3 small Jupyter-related dependencies PR @falexwolf * 🏗️ Pause maintenance of "clinicore" PR @falexwolf Data validation: * ✨ Support validating "attrs" of Pandas DataFrames PR @Zethson import lamindb as ln study_metadata_schema = ln.Schema([ ln.Feature(name="temperature", dtype=float).save(), ln.Feature(name="experiment", dtype=str).save(), ]).save() schema = ln.Schema( slots={"attrs": study_metadata_schema}, # validates keys in df.attrs otype="DataFrame", ).save() * ✨ Support validating linked features: replace "str_as_ulabel" argument with "schema" argument in ".features.add_values()" and enable passing feature annotations to "ln.Artifact" PR PR @Zethson import lamindb as ln schema = ln.Schema([ ln.Feature(name="species", dtype=str).save(), ln.Feature(name="split", dtype=str).save() ]).save() artifact = ln.Artifact( "./test_file.txt", key="test.txt", features={"species": "bird", "split": "train"}, schema=schema, ).save() * ✨ Unstructured slot validation of scverse datastructures PR @Zethson Other features: * ✨ Fine-grained storage access management PR @falexwolf * ✨ Allow getting artifacts by path via "Artifact.get(path="...")" PR @Koncopd * ✨ Add "Feature.from_dict()" PR @Zethson * ✨ Query by feature across instances PR @Koncopd * ✨ Soft delete for all entities PR @falexwolf @Zethson * ✨ Implement lazy artifact saving for streaming data PR @Koncopd * ✨ Support "py:percent" notebooks in "ln.finish()" PR @Koncopd CELLxGENE-compliant curation: * ✨ Enable validating & annotating datasets with a CELLxGENE schema. Corresponding new guide. PR @Zethson * ✨ Support more organisms in CELLxGENE PR @Zethson * ✨ Add support for CELLxGENE Schema 6.0.0 PR @Zethson Bug fixes: * 🐛 Include a project feature in a sheet and test it PR @falexwolf * 🐛 Add missing "RecordUser" link model PR @falexwolf * 🐛 Fix "queryset.to_dataframe(include=...)" with "using" and annotations PR @Koncopd * 🐛 Fix adding a column to a cloud "AnnData" object with consolidated metadata PR @Koncopd * 🐛 Do not reset latest version when deleting latest versions of folder artifacts PR @Koncopd Refactoring: * ♻️ Rename ".list()" to ".to_list()", ".from_df()" to ".from_dataframe()" PR @Koncopd * ♻️ Rename ".df()" to ".to_dataframe()" PR @Koncopd * ♻️ Refactor croissant file mapping PR @falexwolf * ♻️ Refactor and prettify "login()" and "lamin login" PR @falexwolf * ♻️ Add "type" record constraint PR @chaichontat * ♻️ Transfer feature based on uid instead of name PR @sunnyosun * ♻️ Simplify "init_storage()" PR @falexwolf * ♻️ Adapt to new convention for tracking R environments PR @falexwolf UX: * 🚸 Better "lamin delete" and getting versioned entities from trash PR @falexwolf * 🚸 Support "AnnDataAccessor" for "AnnDataCurator" PR @Zethson * 🚸 Migrate easily in presence of fine-grained access connection strings PR @falexwolf * 🚸 Print warning when creating spaces on SQLite or local instances PR @falexwolf * 🚸 Set "connect_timeout" for postgres PR @Koncopd * 🚸 Expose available spaces for a user on fine-grained access instances PR @Koncopd * 🚸 Parse and populate title in R notebooks on "lamin save" PR @Koncopd * 🚸 Allow updating reports of notebooks on "lamin save" PR @Koncopd * 🚸 Expose api url PR @Koncopd Docs: * 📝 Add an MLFlow integration guide PR PR @Zethson * 📝 Improve documentation for "slots" PR @Zethson Upgrades: * ⬆️ Upgrade "fsspec" to 2025.7.0 PR @Koncopd Bionty: * ✨ Add a script to register new standard ontology in "bionty-assets" PR @namsaraeva * ✨ Organism version: ncbitaxon-2025-03-13 PR @Zethson * ✨ New CellType version: cl-2025-04-10 PR @Zethson * ✨ New Tissue version: Uberon-2025-05-28 PR @Zethson * ♻️ Add indexes PR @falexwolf Wetlab: * ✨ Add "smiles" and "inchkey" fields to "Compound" including SMILES normalization PR @azkalot1 Assets: * 🍱 Register CELLxGENE LTS 2025-01-30 registration PR @Zethson Use cases: * 📝 Refactor bulk RNA curation guide to being schema-based PR @Zethson * 📝 Add Croissant integration guide PR @Zethson * ⬆️ Adapt "sc-imaging2" and "facs2" to anndata 0.12 PR @namsaraeva PR @sunnyosun # 2025-08-12 db 1.10.2 * ✨ Write mode for "zarr" backed "AnnDataAccessor" PR @Koncopd * ✨ Add "ProjectRecord" for annotating sheets with projects PR @falexwolf * 🚸 Ask for additional confirmation when creating storage locations through switching storage settings PR @falexwolf * 🐛 Fix "AnnDataAccessor" for "AnnData" objects with indices stored as integers PR @Koncopd * 🐛 Annotate artifacts passed to Curator with Schema PR @Zethson # 2025-08-06 db 1.10.1 * 🐛 Fix selection with a bool mask in "AnnDataAccessor" with "zarr" PR @Koncopd * 🔒️ Require SSL for remote postgres PR @Koncopd * 📝 Add nested dictionary example for Pydantic/Pandera/LaminDB comparison PR @Zethson * 📝 Improve arrays tutorial PR @Koncopd # 2025-07-29 db 1.10.0 Features. * ✨ Enable ".open()" for "SpatialData.tables" PR @Koncopd * ✨ Add "curate_from_croissant()" to curate from MLCommons "Croissant" files PR @falexwolf * ✨ Allow to receive and add extra parameters for managed buckets PR @Koncopd * ✨ Add "Collection.describe()" PR @Zethson * ✨ Enable reverting database migrations PR @falexwolf * ✨ Allow getting settings via the CLI PR @falexwolf UX. * 🚸 Stricter hash uniqueness on artifact and more indexes PR @falexwolf * 🚸 Hide "VitessceConfig" artifacts PR @falexwolf * 🚸 Move ".datasets" from ".core" to ".examples" PR @falexwolf * 🚸 Cache "branch" and "space" settings PR @falexwolf * 🚸 Warn on failure of updating access token PR @Koncopd Refactors and bug fixes. * ♻️ Adapt huggingface sync to the changes in their API PR @Koncopd * 🐛 Fix auto-search for corrupted local storage location PR @falexwolf # 2025-07-23 db 1.9.1 * 🐛 Fix keep-artifacts-local mode when no local storage location is found PR @falexwolf * 🐛 Enable anonymous users to access public folders on AWS S3 PR @Koncopd * 🚸 Create records under subtype via "Curator.add_new_from" PR @sunnyosun * 🚸 On "transform" delete, delete "TransformProject" links because they might be protected through a run of the same transform PR @falexwolf # 2025-07-21 db 1.9.0 | bionty 1.6.1 Features. * ✨ Enable validating & annotating datasets with a CELLxGENE schema. Corresponding new guide. PR @Zethson * ✨ Add "lamin annotate", enable string-based annotation with non- ulabels, overhaul CLI docs PR @falexwolf UX. * 🚸 Enable "ln.track()" and "ln.finish()" for notebooks running on remote servers PR @falexwolf * 🚸 Improve UX of working in "keep-artifacts-local" mode PR @falexwolf * 🚸 Expose "storage", "branch", and "space" in the "Artifact" constructor PR @falexwolf Bug fixes. * 🐛 Fix "search_local_root" in "keep-artifacts-local" for storage roots without access PR @Koncopd * 🐛 Fix sync for cache synchronization for timestamps with a fractional part PR PR @Koncopd * 🐛 Rework "get_storage_region()" to make it reliable and use it in "upath.to_url()" PR @Koncopd * 🐛 Fix creation of a "Schema" with "is_type=True" PR @sunnyosun * 🐛 Fix "index" hash calculation for "Schema" PR @sunnyosun Performance. * ⚡️ Speed up "describe()" by 6x PR @falexwolf * ⚡️ Speed up "lamin connect" by 2x PR @falexwolf * ⚡️ Enable retries for hub requests PR @Koncopd * ⚡️ Implement performant synchronization for directories PR @Koncopd Docs. * 📝 Overhaul the "Curate datasets" guide PR @sunnyosun * 📝 Add an actual "README.md" PR @falexwolf Bionty. * ♻️ New error message when url of ontology doesn't exist PR @namsaraeva * ♻️ Remove Disease constructor overloads PR @namsaraeva * ♻️ Update visibility of "source.dataframe_artifact" PR @sunnyosun * ♻️ Make sure ensembl organism has a "synonyms" column PR @sunnyosun * ♻️ Adapt to renaming of "UPath.synchronize()" to "UPath.synchronize_to()" PR PR @Koncopd * ♻️ Remove pronto warning filters PR @namsaraeva * ♻️ Added logging to "Ontology.to_df()" PR @namsaraeva # 2025-07-14 db 1.8.0 * 🚸 Improve the experience of working in "keep-artifacts-local" mode PR @falexwolf * ⚡️ Use native "polars" Object Store by default PR @Koncopd * 🚸 Improve data type issue errors PR @Zethson * 🚸 Improve error message when attempting to curate against unsaved schema PR @Zethson * 🚸 Improve UX for labeling unsaved records from other instances PR @Zethson * 🐛 Properly ignore "ln.track()" and tracking warnings on read-only connections PR @Koncopd # 2025-07-07 db 1.7.1 * 🚸 Cache "uid" and "name" of "branch" and "space" in setup settings to speed up "lamin info" PR @Koncopd * 🐛 Do not import lamindb within CLI "delete" when deleting an instance PR @Koncopd # 2025-07-06 db 1.7.0 | bionty 1.6.0 -[ ⚠️ Consider "lamin migrate deploy" ]- All instances connected to LaminHub have been migrated and there is no need to act. If you are an admin of a self-managed instance, please migrate your database with "lamin migrate deploy". Necessary database migrations in this release consist in new link tables that can interfere with certain deletion calls. Features. * ✨ Enable switching branches and spaces via the CLI PR @falexwolf * ✨ Enable specifying a system-wide cache path PR @Koncopd * ✨ Upload the R environment tracked in LaminR PR @falexwolf * ✨ Create export functionality for records, enable annotating artifacts by records, enable tracking lineage of records, treat sheets as a record type PR PR PR @falexwolf * ✨ Support "cat_filters" to enable specifying Source versions PR @Zethson * ✨ Correctly deal with renamed instances PR PR @Koncopd * ✨ Add "dtype" "path" PR @sunnyosun UX improvements. * 🚸 Improved storage location management: conveniently create & delete storage locations PR PR PR PR PR @falexwolf @Koncopd * 🚸 Introduce run status codes, save source code & run environment of scripts upon "ln.track()" instead of "ln.finish()" PR @falexwolf * 🚸 Simplify default transfer mode: no longer transfer annotations PR @falexwolf * 🚸 Add "settings.is_connected" to check if the current instance is properly connected for use PR @Koncopd * 🚸 No longer persist instance settings on "ln.connect()" PR @Koncopd * 🚸 Case-insensitive uniqueness and lower-case pre-defined branch and space names PR @falexwolf * 🚸 Warn when no categorical values are validated rather than throwing an error, e.g. when validating column names of a dataframe or labels of a label vector PR @sunnyosun * 🚸 Introduce two-column layout in "Artifact.describe()" and add information like "space", "branch", and "kind" PR @falexwolf * 🚸 Support nested sub types in curation PR PR @sunnyosun @falexwolf * 🚸 Add "Settings" pretty display PR @Zethson * 🚸 Enable inferring all features of a queryset of records or artifacts by passing "features="queryset"" to ".df()" PR @falexwolf * 🚸 Mark data transfers via transform key "__lamindb_transfer__" PR @falexwolf * 🚸 More aggressive checks for "anon_public" requests to S3 for public instances PR @Koncopd * 🚸 Mark upload failures of artifacts via a private boolean indicating a successful save event: "artifact._is_saved_to_storage_location" PR @Koncopd * 🚸 Improve similar names found message PR @Zethson * 🚸 Correct treatment of carriage returns in logs PR @Koncopd * 🚸 Add "keep" with default ""first"" to "Lookup" class PR @sunnyosun * 🚸 Batched bulk saving for persisting large numbers of records efficiently PR @sunnyosun * 🚸 Define all exceptions in "lamindb_setup.errors" PR @falexwolf * 🚸 Leverage panderas lazy validation PR @falexwolf * 🚸 Enable setting "overwrite_versions" in "Artifact" creator PR @falexwolf * 🚸 Default to hiding artifacts with "kind = "__lamindb_run__"" in regular queries PR @falexwolf * 🚸 Mark as "py.typed" to allow mypy inspection and fix return type of "BaseSQLRecord.save" PR PR @ap-- * 🚸 Enable passing "branch" and "space" everywhere PR @falexwolf Bug fixes. * 🐛 Auto-version scripts in absence of "ln.track()" PR @falexwolf * 🐛 Fix updates of re-loaded "Schema" auxiliary fields PR @Zethson * 🐛 Fix a bug preventing backwards compatibility of instance settings files PR @Koncopd * 🐛 Respect "anon=True" in "_tiledb_config_s3" PR @Koncopd * 🐛 Have transfer comply with "settings.annotation.n_max_records" and fix a bug related to repeated schema transfer PR @falexwolf * 🐛 Fix transfer from instance with superset of module to instance with subset of modules in presence of schema-annotation in the additional modules PR @falexwolf Bionty changes. * 🚸 Allow using public references without prefix filters PR @sunnyosun * ⬆️ Add new ontology versions & fix registration bugs PR PR @Zethson * ♻️ Downgrade sources versions in sources.yaml PR @namsaraeva * ♻️ Do not upload dataframe if instance is bionty-assets PR @sunnyosun * ♻️ Make treatment of "Record" link tables consistent with lamindb PR @falexwolf * 🐛 Fix "import_source" with non-existing "parents" in the reference PR @namsaraeva Deprecated code. * 🔥 Remove long-deprecated "ln.setup.load()" and LaminDB v1 migration logic PR @falexwolf # 2025-06-23 nf 0.1.0 * ✨ Track output artifacts and Nextflow runs via transforms and runs PR @rcannood # 2025-06-10 db 1.6.2 * 🐛 Use standard Auth URL to fix API key login (fix **not** necessary after fixing bot detection server side) PR @falexwolf * 🚸 Mark as "py.typed" and improve typing of "ln.setup.connect()" PR @ap-- * 🚸 Expand user directory by default PR @Zethson # 2025-06-03 db 1.6.1 | bionty 1.5.0 Bionty. * ✨ Flexible ontology sources PR @sunnyosun LaminDB. * 🚸 Enable passing "--branch" and "--space" to "lamin save" PR @falexwolf * 🐛 Fix query of feature-associated labels from non-ULabel registries PR @sunnyosun # 2025-06-01 db 1.6.0 | bionty 1.4.0 -[ ⚠️ Consider "lamin migrate deploy" ]- All instances connected to LaminHub have been migrated and there is no need to act. If you are an admin of a self-managed instance, please migrate your database with "lamin migrate deploy". The migrations in this release do *not* break old LaminDB clients with the exception of writing to the "Param" registry: the data in the corresponding SQL table got moved into the "Feature" registry. The bulk of database-level changes was made in this PR @falexwolf. * remove unique constraint from "Feature.name" * replace hard unique constraint on "Transform.hash" and "Artifact.hash", with conditional unique constraint: hash can be duplicated for different keys * new names for how instances are referred to for "type", these don't clash with the new "record" concept: "Ulabel.ulabels", "Feature.features", "Schema.schemas", "Project.projects" * default space uid is now a ""A"" for ""All"" * "Feature._expect_many" now defaults to "None" so that the auto- display of single values as opposed to a set makes sense, and a user can enforce one (single) or the other (many) in the future * hash is populated for all "FeatureValue" records so that there is an easy way to universally identify a unique feature value Changes to registries. * 🏗️ Integrate the "Param" into the "Feature" registry PR @falexwolf -- the change is backward compatible on the Python/R level -- on the SQL level, records are transferred from the "lamindb_param" table to the "lamindb_feature" table during migrations * ✨ Introduce a "Branch" registry PR @falexwolf * ♻️ Rename "Record" to "SQLRecord" PR PR @falexwolf * ✨ Introduce a flexible "Record" registry to manage any kind of entity without database migrations PR @falexwolf Data curation. * ✨ Add schema-based "TiledbsomaExperimentCurator" PR PR @Zethson * ✨ Support curating lists as values in "DataFrameCurator" PR @sunnyosun Bug fixes. * 🐛 Fix transfer for cases in which genes are insufficiently populated PR @falexwolf Dependency changes. * ⬆️ Add compatibility with zarr v3 PR @Koncopd * ⬆️ Add a lower bound on pandera PR PR @Koncopd@Zethson * ⬆️ No longer install "contenttypes" PR @falexwolf UX improvements. * 🚸 Do no longer duplicate tracking of predecessors through the corresponding link table on "Transform" PR @falexwolf * 🚸 Add "is_run_input" to "Artifact.get()" and "Collection.get()" PR @Koncopd * 🚸 Improve suffix mismatch error message PR @Zethson * 🚸 Clearer error in "parse_cat_dtype" if cat dtype contains a module name and the module is not found PR @Koncopd * 🚸 Better error message when user passes manual "uid" to "track()" + anticipate that the user might want to create new transforms in some cases also if hash matches PR @falexwolf * 🚸 Improve setting relationships of unsaved records UX PR @Zethson * 🚸 Improve "DoesNotExist" error message upon "DBRecord.get()" PR @Zethson * ♻️ Set current space when transferring records PR @Koncopd * ♻️ Mark internal lamindb-produced artifacts with "kind="__lamindb__"" instead of "_branch_code=0" PR @falexwolf # 2025-05-13 db 1.5.3 * 🐛 Fix schema transfer PR @Zethson # 2025-05-13 db 1.5.2 * 🐛 Reset "SpatialData" path when access in-memory representation PR @Zethson * 🚸 Do not validate twice within "Artifact.from_X(...)" when passing schema PR @falexwolf # 2025-05-08 db 1.5.1 * 🐛 Fix a too strict unique constraint in composite schemas PR @falexwolf * 🐛 Fix display of parents & children in "view_parents(with_children)" PR @falexwolf * ⬆️ Adapt "save_tiledbsoma_experiment" to "tiledbsoma==1.16.2" PR @Koncopd # 2025-05-07 db 1.5.0 | bionty 1.3.2 Data lineage. * 🚸 Make notebook & script tracking via "ln.track()" robust to renames PR @falexwolf * ✨ Enable executing notebooks via "jupyter nbconvert --execute" PR @falexwolf CLI updates. * ✨ Enable cloud paths for "lamin save" PR @Koncopd lamin save s3://my-bucket/my-file.txt * ✨ Enable labeling with project during "lamin save" PR @falexwolf lamin save ./my-folder --project my-project Streaming artifacts. * ✨ Enable "polars" in "Artifact.open()" and "Collection.open()" PR @Koncopd * ✨ Enable ".load()", ".open()", and ".mapped()" on query sets of artifacts PR @Koncopd Curation & schemas. * ✨ Enable curating the index of a dataframe PR @falexwolf schema = ln.Schema( features=[ ln.Feature(name="required_feature", dtype=str).save(), ], index=ln.Feature(name="sample", dtype=ln.ULabel).save(), ).save() * 🚸 Enable passing a "ULabel" type to "dtype" PR @falexwolf perturbation_type = ln.ULabel.get(name="Perturbation") # perturbation_type.is_type is True ln.Feature(name="perturbation", dtype=perturbation_type) * 🚸 Handle schema updates decently PR @falexwolf * 🚸 Do not annotate with more than "n_max_records = 1000" PR @falexwolf * 🚸 Introduce a submodule "lamindb.examples" with schemas PR @falexwolf * 🚸 Enable validating against nested dicts in "spatialdata" PR @falexwolf * 🚸 Better handle validation of ensembl gene IDs and add curator representation PR @Zethson * 🚸 Prettier "Schema.describe()" PR @sunnyosun * 🚸 "AnnData": enable explicit transposition in "var" schema definition PR @falexwolf * 🚚 Rename the "components" argument of "Schema()" to "slots" PR @falexwolf * 🐛 Fix respecting "schema.ordered_set" in "DataFrame" validation PR @sunnyosun Bulk annotation with features & queries via features. * ✨ Support feature dtype "dict" PR @falexwolf ln.Feature(name="metadata_details", dtype=dict).save() * 🚸 For artifacts, improve (1) bulk annotation with features + (2) queries by features PR @falexwolf General UX improvements. * 🚸 Do not raise exceptions on problems with "copy_or_move_to_cache" within "Artifact.save" PR @Koncopd * 🚸 Allow passing "key" to "save_vitessce_config()" PR @namsaraeva * 🚸 Add ".pt" and ".ckpt" to valid suffixes PR @Zethson Docs. * 📝 Document "uid" generation, prettify API reference docs PR @falexwolf -[ Refactoring. ]- General refactoring. * ♻️ Eliminate monkey patching of "django.db.models.QuerySet" and "django.db.models.Manager" PR @Koncopd * ♻️ Avoid non-lazy loads of settings on import of "lamindb.models" PR @Koncopd Refactoring for curation & schemas. * ♻️ Restore validation error messages & add their fine-grained testing PR @falexwolf * ♻️ Can save csv artifacts in "DataFrameCurator" PR @sunnyosun * ♻️ Clearer naming conventions in the internal curator codebase PR @falexwolf * ♻️ Separate "CatManager" usage for ".cat" attribute and as legacy interface PR @falexwolf * ♻️ Separate legacy curators from new curators PR @falexwolf * ♻️ Execute curator examples and also show them in the curation guide PR @falexwolf * ♻️ Refactor annotating with inferred feature sets PR @falexwolf -[ Fine-grained access management (in beta). ]- * 🚸 Better access management errors on "Record.save()" PR @Koncopd * ✨ Refresh db token on expiration PR @Koncopd * 🐛 Fix ".using" with fine-grained access instances and permissions test PR @Koncopd * ✅ Test auth errors PR @Koncopd * ✅ Temp table based authentication (adapt tests) PR @Koncopd * ✅ Decrypt token inside RLS (adapt tests) PR @Koncopd * 🚸 Delete version family if user wants to retain store by passing "storage=False" to "artifact.delete()", but retain warning PR @falexwolf # 2025-04-25 bionty 1.3.1 🐛 Fixed downloading old Ensembl versions. PR @sunnyosun -[ If you upgraded to "bionty" 1.3.0 and used Ensembl versions below 108, please clear the cached ontology source files. ]- import bionty as bt import shutil shutil.rmtree(bt.base.settings.dynamicdir) # 2025-04-24 R 1.1.0 -[ LaminR is now documented on "docs.lamin.ai". ]- The previous docs site "laminr.lamin.ai" continues to host developer docs. * 📝 Update documentation site to match the main docs website PR PR @lazappi * 👷 Separate Seurat analysis from rest of the introduction notebook PR @falexwolf * ♻️ Make R and Python quickstarts parallel PR @falexwolf * ♻️ Move "setup.Rmd" to "lamin-docs" PR @falexwolf New features. * ✨ Improved Python dependency management with "reticulate", deprecated "install_lamindb()" PR @lazappi * ✨ Add tracking of the R environment using "pak" lockfiles PR @lazappi * ✨ Enable "artifact$view_lineage()" PR @lazappi Bug fixes. * 🐛 Enable setting wrapped object slots like "artifact$description", "artifact$key", etc. PR @lazappi * 🐛 Fix an issue that was preventing "lamin_connect()" from being run multiple times with the same instance PR @lazappi * 🐛 Properly clear and delete temporary instances created using "lamin_init_temp()" PR @lazappi Other changes. * ♻️ Set minimum "reticulate" dependency ">= 1.38.0" PR @lazappi * 🚸 Improve inheritance of arguments when wrapping and overwriting Python functions PR @lazappi * ♻️ Dispatch CI from pre-release events in lamindb PR @falexwolf # 2025-04-15 db 1.4.0 | bionty 1.3.0 ✨ Add "schema" as an argument to "Artifact.from_X()". PR @falexwolf artifact = ln.Artifact.from_df(df, key="my_dataset.parquet", schema=schema).save() ✨ Enable defining simple schemas that merely enforce a feature identifier type. PR @falexwolf schema = ln.Schema(itype=ln.Feature).save() # <-- enforce valid feature identifiers, no need to define specific required features ✨ Enable defining optional features on a per-schema level & improve schema hash calculation. PR @sunnyosun schema = ln.Schema( features=[ ln.Feature(name="sample_id", dtype=str).save() # required ln.Feature(name="sample_name", dtype=str).with_config(optional=True) # optional ], ).save() ✨ Introduce "lamin run" with a Modal backend. PR @ragyhaddad lamin run my_script.py --project my_project # <-- will run the script on Modal ✨ Support auto-download of Ensembl genes of all organisms. PR @sunnyosun gene_ontology = bt.base.Gene(source="ensembl", organism="rabbit", version='release-103') gene_ontology.register_source_in_lamindb() # register the new ontology source in lamindb source = bt.Source.get(entity="bionty.Gene", name="ensembl", organism="rabbit", version='release-103') bt.Gene.import_source(source=source) # import all genes from that source 🚸 Enable querying by features & params through "Artifact.filter()" and "Run.filter()". PR @falexwolf ln.Artifact.filter(scientist="Barbara McClintock") User experience. * 🚸 "from_source" no longer returns "None" but throws a "NoResultFound" exception if the look up in the public ontology fails PR @sunnyosun * 🚸 Allow renaming artifacts & transforms within the same version family PR @falexwolf * 🚸 Better support "minimal_set", "maximal_set", "ordered_set" in curators PR @sunnyosun * 🚸 Enable passing the stem uid to "lamin save" PR @falexwolf * 🚸 No longer throw an error but merely print a warning when attempting to update a schema PR @falexwolf * 🚸 Enable plain notebook uploads by making a default run for notebook in case no run is found PR @falexwolf * 🚸 Enable to authenticate and set the current instance through environment variables PR @falexwolf * 🚸 Show link to hub in "view_lineage()" and render lineage through graphviz also in scripts PR @falexwolf * 🚸 Order "IsVersioned.versions" query set PR @falexwolf * 🚸 Do not print warning about missing schema modules PR @falexwolf Refactors. * ♻️ Eliminate duplicated parsing & record creation during curation PR @falexwolf * ♻️ Remove "verbosity" and "organism" arguments on "CatManager" level PR PR @falexwolf * ♻️ Organize categorical curation code with "CatColumn" PR @sunnyosun * ♻️ Add "return_graph" argument to "view_lineage()" PR @lazappi * ♻️ Suppress "aiobotocore" traceback logging PR @Koncopd * ⬆️ Upgrade "supabase" to "<2.15.0" PR @Koncopd * ⬆️ Upgrade "anndata" to "0.11.4" PR @Koncopd Docs. * 📝 Compare lamindb with pydantic and pandera in an FAQ doc PR @falexwolf * 📝 Document access any Ensembl genes PR @sunnyosun Bugs. * 🐛 Fix validation of "var_index" PR @sunnyosun * 🐛 Fix "numcodecs==0.16.0" incompatibility with "zarr v2" PR @Koncopd * 🐛 Fix "SpatialData" and "MuData" check PR @Zethson * 🐛 Fix organism passing to "from_source" PR @sunnyosun * 🐛 Return an empty set not a dict for modules in instance settings PR @falexwolf Bionty. * 🚸 Make the default organism ""human"" instead of "None" PR @falexwolf * ⬆️ Support Python 3.13 & remove support for Python 3.9 PR @Zethson * ♻️ Improve Ensembl prefix detection PR @sunnyosun * ♻️ Use "UPath.synchronize" in "s3_bionty_assets" PR @Koncopd # 2025-03-27 db 1.3.2 | bionty 1.2.1 * 🐛 Fix bionty ontology sources sync through "reticulate" PR @falexwolf * 🐛 Fix data transfer through when target instances has no schema modules PR @falexwolf # 2025-03-26 db 1.3.1 | bionty 1.2.0 In Bionty, you can now add custom ontology sources through the "Source" registry. df = pd.read_csv("./our_inhouse_genes.csv") # a csv describing gene metadata e.g. from parsing a GTF file custom_source = bt.Source(entity="bionty.Gene", organism="human", name="Our genes", version="2025-04-01").save() bt.Gene.add_source(custom_source, df=df) # couple the custom source to the Gene registry -[ Detailed changes ]- Bionty now relies on a single file source.yaml to reference public sources. * ✨ Enable update existing records to a new ontology PRPR @sunnyosun * ✨ Robust support of custom sources PR @sunnyosun * ➖ Remove pronto from main dependencies PR @Zethson * ♻️ Refactor "sync_public_sources" PR @sunnyosun * ♻️ Refactor default source configuration PR @sunnyosun * ♻️ Make EFO parsing the same as other ontologies PR @sunnyosun * ♻️ No longer use local source yaml files PR @sunnyosun * ♻️ Move source tests from lamindb to bionty PR @sunnyosun * ♻️ Standardize organism scientific names from ensembl source PR @sunnyosun * ♻️ Increase uid length for "Source" to 8 chars PR @falexwolf * 🍱 New "ExperimentalFactor" version: "efo-3.69.0" PR @Zethson * 🍱 New "CellType" version: "cl"-"2024-08-16" PR @Zethson * 🍱 New "Disease" version: "mondo"-"2024-08-06" PR @Zethson LaminDB changes. * 🐛 Fix incompatibility with "gotrue==2.12.0" PR @Koncopd * 🐛 Enable transferring features pointing to multiple labels PR @sunnyosun * 🐛 More extensive validation for updates to "artifact.key" and "artifact.suffix" PR @falexwolf * 🚸 Refactor conventions for files written during init: the SQLite file is now ".lamindb/lamin.db" and the storage marker is ".lamindb/storage_uid.txt" PR @falexwolf * 🚸 Make upload of large directories more robust by reducing batch size PR @Koncopd * 🚸 Avoid requiring "coerce_dtype" for ""int"" and ""float"" in case an integer or float "pd.Series.dtype" only deviates by numerical precision/range PR @falexwolf * 🚸 In "AnnDataCurator", make "'obs'" schema optional and allow "'uns'" schema PR @falexwolf # 2025-03-16 db 1.3.0 New features. * ✨ Add schema-based "SpatialDataCurator" PR1 PR2 PR3 @Zethson * ✨ Add schema-based "MuDataCurator" PR @sunnyosun * ✨ Add "lamin get" for artifacts and "lamin load" for collections PR @Zethson @falexwolf Other changes. * ⬆️ Python 3.13 support PR @Zethson * ⬆️ Support CELLxGENE schema 5.2.0 PR1 PR2 @sunnyosun * 🚸 Skip "ln.track()" when connected in read-only mode PR @falexwolf * 🚸 Error if trying to register an instance without a storage in the hub PR @Koncopd * 🚸 Refactor "organism" constraints during validation PR @sunnyosun * 🚸 Add more constructor signatures and specific inherited types PR @falexwolf * 🚸 No logging message if database is behind by minor version PR @falexwolf * 📝 Re-structure curation guides PR1 PR2 @falexwolf * 📝 Integrate tutorials into introduction guide PR @falexwolf # 2025-03-10 R 1.0.0 ✨ "laminr" now has feature parity with "lamindb". PR @lazappi * Run "install_lamindb()", which will ensure "lamindb >= 1.2" in the Python environment used by "reticulate". * Replace "db <- connect()" with "ln <- import_module("lamindb")" and see the "Detailed changes" dropdown. The "ln" object is largely similar to the "db" object in "laminr" < v1 and matches "lamindb"'s Python API ("." → "$"). -[ Detailed changes ]- | --- | --- | --- | | What | Before | After | | =================================== | =================================== | =================================== | | Connect to the default LaminDB | "db <- connect()" | "ln <- import_module("lamindb")" | | instance | | --- | --- | --- | | Start tracking | "db$track()" | "ln$track()" | | --- | --- | --- | | Get an artifact from another | "new_instance <- | "ln$Artifact$using("another/inst | | instance | connect("another/instance"); | ance")$get(...)" | | new_instance$Artifact$get(...)" | | --- | --- | --- | | Create an artifact from a path | "db$Artifact$from_path(path)" | "ln$Artifact(path)" | | --- | --- | --- | | Finish tracking | "db$finish()" | "ln$finish()" | | --- | --- | --- | See the updated "Get started" vignette for more information. User-facing changes: * Add an "import_module()" function to import Python modules with additional functionality, e.g., "import_module("lamindb")" for **lamindb** * Add functions for accessing more "lamin" CLI commands * Add a new "Introduction" vignette that replicates the code from the Python **lamindb** introduction guide Internal changes: * Add an internal "wrap_python()" function to wrap Python objects while replacing Python methods with R methods as needed, leaving most work to **{reticulate}** * Update the internal "check_requires()" function to handle Python packages * Add custom "cache()"/"load()" methods to the "Artifact" class * Add custom "track()"/"finish()" methods to the **lamindb** module # 2025-03-09 db 1.2.0 ✨ Enable to auto-link entities to projects. PR @falexwolf ln.track(project="My project") 🚸 Better support for "spatialdata" with "Artifact.from_spatialdata()" and "artifact.load()". PR1 PR2 @Zethson 🚸 Introduce ".slots" in "Schema", "Curator", and "artifact.features" to access schemas and curators by dataset slot. PR @sunnyosun schema.slots["obs"] # -> schema for .obs slot of AnnData curator.slots["obs"] # -> curator for .obs slot of AnnData artifact.features["obs"] # -> feature set for .obs slot of AnnData 🏗️ Re-structured the internal API away from monkey-patching Django models. PR @falexwolf -[ ⚠️ Use of internal API ]- If you used the internal API, you might experience a breaking change. The most drastic change is that all internal registry-related functionality is now re-exported under "lamindb.models". 🚸 When re-creating an "Artifact", link subsequent runs instead of updating ".run" and linking previous runs. PR @falexwolf -[ On the hub. ]- More details here. @chaichontat | --- | --- | | Before | After | | ==================================================== | ==================================================== | | An artifact is only shown as an output for the | All runs that (re-)create an artifact show it as | | *latest run* that created the artifact. *Previous* | an output. | | runs don't show it. | | --- | --- | | --- | --- | | --- | --- | More changes: * ✨ Support R2 PR @Koncopd * ✨ Enable "Artifact.open()" and "Artifact.load()" for ".gz" files PR @Koncopd * 🐛 Fix passing a path to "ln.track()" when no path found by "nbproject" PR @Koncopd * 🐛 Do not overwrite "._state_db" of records when the current instance is passed to ".using" PR @Koncopd * 🚸 Do not show track warning for read-only connections PR @Koncopd * 🚸 Raise "NotImplementedError" in "Artifact.load()" if there is no loader PR @Koncopd # 2025-02-27 db 1.1.1 * 🚸 Make the "obs" and "var" "DataFrameCurator" objects accessible via "AnnDataCurator.slots" PR @sunnyosun * 🚸 Better error message upon re-creation of schema with same name and different hash PR @falexwolf * 🚸 Raise consistency error if a source path suffix doesn't match the artifact "key" suffix PR @falexwolf * 🚸 Automatically add missing columns upon "DataFrameCurator.standardize()" if "nullable" is "True" PR @falexwolf * 🚸 Allow specifying "fsspec" upload options in "Artifact.save" PR @Koncopd * 🚸 Populate "Artifact.n_observations" in "Artifact.from_df()" PR @Koncopd * 🐛 Fix "UPath.view_tree" on first call on "gs" PR @Koncopd * 🐛 Fix ".add_new_from" message PR @Zethson * 🐛 Run "pip freeze" with current python interpreter PR @ap-- * 🐛 Do not resolve http links when registering PR @Koncopd * 🐛 Fix notebook re-run with same hash PR @falexwolf # 2025-02-18 db 1.1.0 -[ ⚠️ The "FeatureSet" registry got renamed to "Schema". ]- All your code is backward compatible. The "Schema" registry encompasses feature sets as a special case. ✨ Conveniently track functions including inputs, outputs, and parameters with a decorator: "ln.tracked()". PR1 PR2 @falexwolf @ln.tracked() def subset_dataframe( input_artifact_key: str, # all arguments tracked as parameters of the function run output_artifact_key: str, subset_rows: int = 2, subset_cols: int = 2, ) -> None: artifact = ln.Artifact.get(key=input_artifact_key) df = artifact.load() # auto-tracked as input new_df = df.iloc[:subset_rows, :subset_cols] ln.Artifact.from_df(new_df, key=output_artifact_key).save() # auto-tracked as output ✨ Make sub-types of "ULabel", "Feature", "Schema", "Project", "Param", and "Reference". PR @falexwolf -[ On the hub. ]- More details here. @awgaan @chaichontat | --- | --- | | Before | After | | ==================================================== | ==================================================== | | --- | --- | | --- | --- | perturbation = ln.ULabel(name="Perturbation", is_type=True).save() ln.ULabel(name="DMSO", type=perturbation).save() ln.ULabel(name="IFNG", type=perturbation).save() ✨ Use an overhauled dataset curation flow. @falexwolf @Zethson @sunnyosun * support persisting validation constraints as a "pandera"-compatible schema * support validating any feature type, no longer just categoricals * make the relationship between features, dataset schema, and curator evident -[ Detailed changes for the overhauled curation flow. ]- ⚠️ The API gained the "lamindb.curators" module as the new way to access "Curator" classes for different data structures. * This release introduces the schema-based "DataFrameCurator" and "AnnDataCurator" * The old-style curation flow for categoricals based on "lamindb.Curator.from_objecttype()" continues to work | --- | --- | | Before | After | | ==================================================== | ==================================================== | | --- | --- | | --- | --- | | --- | --- | | --- | --- | Key PRs. * ✨ Overhaul curation guides + enable default values and filters on valid categories for features PR @falexwolf * ✨ Schema-based curators: "AnnDataCurator" PR @falexwolf * ✨ Schema-based curators: "DataFrameCurator" PR @falexwolf Enabling PRs. * ✨ Allow passing "artifact" to "Curator" PR @sunnyosun * 🎨 A "ManyToMany" between "Schema.components" and ".composites" PR @falexwolf * ♻️ Mark "Schema" fields as non-editable PR @falexwolf * ✨ Add auxiliary field "nullable" to "Feature" PR @falexwolf * ♻️ Prettify "AnnDataCurator" implementation PR @falexwolf * 🚸 Better error for malformed categorical dtype PR @falexwolf * 🎨 A "ManyToMany" between "Schema.components" and ".composites" PR @falexwolf * 🚚 Restore ".feature_sets" as a "ManyToManyField" PR @falexwolf * 🚚 Rename "CatCurator" to "CatManager" PR @falexwolf * 🎨 Let "Curator.validate()" throw an error PR @falexwolf * ♻️ Re-purpose "BaseCurator" as "Curator", introduce "CatCurator" and consolidate shared logic under "CatCurator" PR @falexwolf * ♻️ Refactor "organism" handling in curators PR @falexwolf * 🔥 Eliminate all logic related to "using_key" in curators PR @falexwolf * 🚚 Bulk-rename old-style curators to "CatCurator" PR @falexwolf * 🎨 Self-contained definition of "CellxGene" schema / validation constraints PR @falexwolf * 🚚 Move "PertCurator" from "wetlab" here and add "CellxGene" "Curator" test PR @falexwolf * 🚚 Move CellXGene "Curator" from "cellxgene-lamin" here PR @falexwolf schema = ln.Schema( name="small_dataset1_obs_level_metadata", features=[ ln.Feature(name="CD8A", dtype=int).save(), # integer counts for CD8A marker ln.Feature(name="perturbation", dtype=ln.ULabel).save(), # a categorical feature that validates against the ULabel registry ln.Feature(name="sample_note", dtype=str).save(), # a note for the sample ], ).save() df = pd.DataFrame({ "CD8A": [1, 4, 0], "perturbation": ["DMSO", ], "sample_note": ["value_1", "value_2", "value_3"], "temperature": [22.2, 25.7, 27.3], }) curator = ln.curators.DataFrameCurator(df, schema) artifact = curator.save_artifact(key="example_datasets/dataset1.parquet") # validates compliance with schema, annotates with metadata assert artifact.schema == schema # the validating schema ✨ Easily filter on a validating schema. @falexwolf @Zethson @sunnyosun -[ On the hub. ]- With the "Schema" filter button, find all datasets that satisfy a given schema (→ explore). schema = ln.Schema.get(name="small_dataset1_obs_level_metadata") # get a schema ln.Artifact.filter(schema=schema).df() # filter all datasets that were validated by the schema ✨ "Collection.open()" returns a "pyarrow" dataset. PR @Koncopd df = pd.DataFrame({"feat1": [0, 0, 1, 1], "feat2": [6, 7, 8, 9]}) df[:2].to_parquet("df1.parquet", engine="pyarrow") df[2:].to_parquet("df2.parquet", engine="pyarrow") artifact1 = ln.Artifact(shard1, key="df1.parquet").save() artifact2 = ln.Artifact(shard2, key="df2.parquet").save() collection = ln.Collection([artifact1, artifact2], key="parquet_col") dataset = collection.open() # backed by files in the cloud storage dataset.to_table().to_pandas().head() ✨ Support s3-compatible endpoint urls, say your on-prem MinIO deployment. PR @Koncopd Speed up instance creation through squashed migrations. * ⚡ Squash migrations PR1 PR2 @falexwolf Tiledbsoma. * ✨ Support "endpoint_url" in operations with tiledbsoma PR1 PR2 @Koncopd * ✨ Add "Artifact.from_tiledbsoma" to populate "n_observations" PR @Koncopd MappedCollection. * 🐛 Allow filtering on "np.nan" in "obs_filter" of "MappedCollection" PR @Koncopd * 🐛 Fix labels for "NaN" in categorical columns for "MappedCollection" PR @Koncopd SpatialDataCurator. * 🐛 Fix "var_index" standardization of "SpatialDataCurator" PR1 PR2 @Zethson * 🐛 Fix sample level metadata optional in "SpatialDataCatManager" PR @Zethson Core functionality. * ✨ Allow checking the need for syncing without actually syncing PR @Koncopd * ✨ Check for corrupted cache in "Artifact.load()" & "Artifact.open()" PR PR @Koncopd * ✨ Infer "n_observations" in "Artifact.from_anndata" PR @Koncopd * 🐛 Account for VSCode appending languageid to markdown cell in notebook tracking PR @falexwolf * 🐛 Fix dangling folders on upload failures PR @Koncopd * 🐛 Normalize module names for robust checking in "_check_instance_setup()" PR @Koncopd * 🐛 Fix idempotency of "Feature" creation when "description" is passed and improve filter and get error behavior PR @Zethson * 🐛 Fix caching logic in "Artifact.open()" PR @Koncopd * 🚸 Make new version upon passing existing "key" to "Collection" PR @falexwolf * 🚸 Throw better error upon checking "instance.modules" when loading a lamindb schema module PR @Koncopd * 🚸 Validate existing records in the DB irrespective of whether an ontology "source" is passed or not PR @sunnyosun * 🚸 Full guarantee of avoiding duplicating "Transform", "Artifact" & "Collection" in concurrent runs PR @falexwolf * 🚸 Fix "RemovedInDjango60Warning" PR @Zethson * 🚸 Better user feedback during keyword validation in "Record" constructor PR @Zethson * 🚸 Fix warning about artifacts in trash PR @ap-- * 🚸 Improved error message when saving via CLI PR @Zethson * 🚸 Improve local storage not found warning message PR @Zethson * 🚸 Better error message when attempting to save a file while not being connected to an instance PR @Zethson * 🚸 Error for non-keyword parameters for "Artifact.from_x" methods PR @Zethson Housekeeping. * 🚸 Error at runtime with old s3fs PR @Koncopd * 🚸 Safer resolve in "check_path_is_child_of_root()" PR @Koncopd * ⬆️ Upgrade fsspec packages (s3fs, gcsfs, universal_pathlib) PR @Koncopd * ➕ Add pyyaml to dependencies PR @Koncopd # 2025-01-23 db 1.0.5 * 🚸 No longer throw a "NotebookNotSaved" error in "ln.finish()" but wait for the user or gracefully exit PR @falexwolf * 🚸 Resolve save "FutureWarning" PR @Zethson * 🐛 Fix "Artifact.replace()" for folder-like artifacts PR @Koncopd * 🐛 Filter the latest transform on saving by filename PR @Koncopd # 2025-01-21 db 1.0.4 🚚 Revert "Collection.description" back to unlimited length "TextField". PR @falexwolf # 2025-01-21 db 1.0.3 🚸 In "track()", improve logging in RStudio sessions. PR @falexwolf # 2025-01-20 R 0.4.0 * 🚚 Migrate to lamindb v1 PR @falexwolf * 🚸 Improve the user experience for setting up Python & reticulate PR @lazappi # 2025-01-20 db 1.0.2 🚚 Improvments for lamindb v1 migrations. PR @falexwolf * add a ".description" field to "Schema" * enable labeling "Run" with "ULabel" * add a ".predecessors" and ".successors" field to "Project" akin to what's present on "Transform" * make ".uid" fields not editable # 2025-01-18 db 1.0.1 🐛 Block non-admin users from confirming the dialogue for integrating "lnschema-core". PR @falexwolf # 2025-01-17 db 1.0.0 This release makes the API consistent, integrates "lnschema_core" & "ourprojects" into the "lamindb" package, and introduces a breadth of database migrations to enable future features without disruption. You'll now need at least **Python 3.10**. -[ Your code will continue to run as is, but you will receive warnings about a few renamed API components. ]- | --- | --- | --- | | What | Before | After | | =================================== | =================================== | =================================== | | Dataset vs. model | "Artifact.type" | "Artifact.kind" | | --- | --- | --- | | Python object for "Artifact" | "Artifact._accessor" | "Artifact.otype" | | --- | --- | --- | | Number of files | "Artifact.n_objects" | "Artifact.n_files" | | --- | --- | --- | | "name" arg of "Transform" | "Transform(name="My notebook", | "Transform(key="my- | | key="my-notebook.ipynb")" | notebook.ipynb", description="My | | notebook")" | | --- | --- | --- | | "name" arg of "Collection" | "Collection(name="My | "Collection(key="My collection")" | | collection")" | | --- | --- | --- | | Consecutiveness field | "Run.is_consecutive" | "Run._is_consecutive" | | --- | --- | --- | | Run initiator | "Run.parent" | "Run.initiated_by_run" | | --- | --- | --- | | "--schema" arg | "lamin init --schema | "lamin init --modules | | bionty,wetlab" | bionty,wetlab" | | --- | --- | --- | Migration guide: 1. Upon "lamin connect account/instance" you will be prompted to confirm migrating away from "lnschema_core" 2. After that, you will be prompted to call "lamin migrate deploy" to apply database migrations New features: * ✨ Allow http storage backend for "Artifact" PR @Koncopd * ✨ Add "SpatialDataCurator" PR @Zethson * ✨ Allow filtering by multiple "obs" columns in "MappedCollection" PR @Koncopd * ✨ In git sync, also search git blob hash in non-default branches PR @Zethson * ✨ Add relationship with "Project" to everything except "Run", "Storage" & "User" so that you can easily filter for the entities relevant to your project PR @falexwolf * ✨ Capture logs of scripts during "ln.track()" PR1 PR2 @falexwolf @Koncopd | * ✨ Support "" | ""-seperated multi-values in "Curator" PR @sunnyosun | * 🚸 Accept "None" in "connect()" and improve migration dialogue PR @falexwolf UX improvements: * 🚸 Simplify the "ln.track()" experience PR @falexwolf 1. you can omit the "uid" argument 2. you can organize transforms in folders 3. versioning is fully automated (requirement for 1.) 4. you can save scripts and notebooks without running them (corollary of 1.) 5. you avoid the interactive prompt in a notebook and the throwing of an error in a script (corollary of 1.) 6. you are no longer required to add a title in a notebook * 🚸 Raise error when modifying "Artifact.key" in problematic ways PR1 PR2 @sunnyosun @Koncopd * 🚸 Better error message on running "ln.track()" within Python terminal PR @Koncopd * 🚸 Hide traceback for "InstanceNotEmpty" using Click Exception PR @Zethson * 🚸 Hide underscore attributes in "__repr__" PR @Zethson * 🚸 Only auto-search "._name_field" in sub-classes of "CanCurate" PR @falexwolf * 🚸 Simplify installation & API overview PR @falexwolf * 🚸 Make "lamin_run_uid" categorical in tiledbsoma stores PR @Koncopd * 🚸 Add defensive check for "organism" arg PR @Zethson * 🚸 Raise "ValueError" when trying to search a "None" value PR @Zethson Bug fixes: * 🐛 Skip deleting storage when deleting outdated versions of folder- like artifacts PR @Koncopd * 🐛 Let "SOMACurator()" validate and annotate all ".obs" columns PR @falexwolf * 🐛 Fix renaming of feature sets PR @sunnyosun * 🐛 Do not raise an exception when default AWS credentials fail PR @Koncopd * 🐛 Only map synonyms when field is name PR @sunnyosun * 🐛 Fix "source" in ".from_values" PR @sunnyosun * 🐛 Fix creating instances with storage in the current local working directory PR @Koncopd * 🐛 Fix NA values in "Curator.add_new_from()" PR @sunnyosun Refactors, renames & maintenance: * 🏗️ Integrate "lnschema-core" into "lamindb" PR1 PR2 @falexwolf @Koncopd * 🏗️ Integrate "ourprojects" into lamindb PR @falexwolf * ♻️ Manage "created_at", "updated_at" on the database-level, make "created_by" not editable PR @falexwolf * 🚚 Rename transform type "glue" to "linker" PR @falexwolf * 🚚 Deprecate the "--schema" argument of "lamin init" in favor of "-- modules" PR @falexwolf * ⬆️ Compatibility with tiledbsoma==1.15.0 PR @Koncopd DevOps: * 👷 Isolate curator tests PR @Zethson -[ Detailed list of database migrations ]- Those not yet announced above will be announced with the functionality they enable. * ♻️ Add "contenttypes" Django plugin PR @falexwolf * 🚚 Prepare introduction of persistable "Curator" objects by renaming "FeatureSet" to "Schema" on the database-level PR @falexwolf * 🚚 Add a ".type" foreign key to "ULabel", "Feature", "FeatureSet", "Reference", "Param" PR @falexwolf * 🚚 Introduce "RunData", "TidyTable", and "TidyTableData" in the database PR @falexwolf All remaining database schema changes were made in this PR @falexwolf. Data migrations happen automatically. * remove "_source_code_artifact" from Transform, it's been deprecated since 0.75 * data migration: for all transforms that have "_source_code_artifact" populated, populate "source_code" * rename "Transform.name" to "Transform.description" because it's analogous to "Artifact.description" * backward compat: * in the "Transform" constructor use "name" to populate "key" in all cases in which only "name" is passed * return the same transform based on "key" in case "source_code is None" via "._name_field = "key"" * data migrations: * there already was a legacy "description" field that was never exposed on the constructor; to be safe, we concatenated potential data in it on the new description field * for all transforms that have "key=None" and "name!=None", use "name" to pre-populate "key" * rename "Collection.name" to "Collection.key" for consistency with "Artifact" & "Transform" and the high likelihood of you wanting to organize them hierarchically * a "_branch_code" integer on every record to model pull requests * include "visibility" within that code * repurpose "visibility=0" as "_branch_code=0" as "archive" * put an index on it * code a "draft" as _branch_code = 2, and "draft prs" as negative branch codes * rename values ""number"" to ""num"" in dtype * an "._aux" json field on "Record" * a SmallInteger "run._status_code" that allows to write "finished_at" in clean up operations so that there is a run time also for aborted runs * rename "Run.is_consecutive" to "Run._is_consecutive" * a "_template_id" FK to store the information of the generating template (whether a record is a template is coded via _branch_code) * rename "_accessor" to "otype" to publicly declare the data format as "suffix, accessor" * rename "Artifact.type" to "Artifact.kind" * a FK to artifact "run._logfile" which holds logs * a "hash" field on "ParamValue" and "FeatureValue" to enforce uniqueness without running the danger of failure for large dictionaries * add a boolean field "._expect_many" to "Feature"/"Param" that defaults to "True"/"False" and indicates whether values for this feature/param are expected to occur a single or multiple times for every single artifact/run * for feature * if it's "True" (default), the values come from an observation- level aggregation and a dtype of "datetime" on the observation- level mean "set[datetime]" on the artifact-level * if it's "False" it's an artifact-level value and "datetime" means "datetime"; this is an edge case because an arbitrary artifact would always be a set of arbitrary measurements that would need to be aggregated ("one just happens to measure a single cell line in that artifact") * for param * if it's "False" (default), the values mean artifact/run-level values and "datetime" means "datetime" * if it's "True", the values would be from an aggregation, this seems like an edge case but say when characterizing a model ensemble trained with different parameters it could be relevant * remove the ".transform" foreign key from artifact and collection for consistency with all other records; introduce a property and a simple filter statement instead that maintains the same UX * store provenance metadata for "TransformULabel", "RunParamValue", "ArtifactParamValue" * enable linking projects & references to transforms & collections * rename "Run.parent" to "Run.initiated_by_run" * introduce a boolean flag on artifact that's called "_overwrite_versions", which indicates whether versions are overwritten or stored separately; it defaults to "False" for file- like artifacts and to "True" for folder-like artifacts * Rename "n_objects" to "n_files" for more clarity * Add a "Space" registry to lamindb with an FK on every "BasicRecord" * add a name column to "Run" so that a specific run can be used as a named specific analysis * remove "_previous_runs" field on everything except "Artifact" & "Collection"