Unrestricted metadata flags

Unrestricted metadata flags#

Theme: Data

DISCLAIMER: Please note that the output of the code in the cells below might change over time. This can happen because Python packages and their dependencies change due to updates, or because new data is added.

This module provides an introduction to metadata flags that have been added recently to the vector resources. These flags are used to filter samples that are “unrestricted”, i.e., available for anyone to use freely. What these terms mean will be examined in more details shortly.

This module will not contain a lot of technical code but some functions, such as sample_metadata and cnv_hmm will be used as examples. We will not detail what these functions do in this notebook and we will not be interested in the results of their executions, only on the impact of using the two flags on these functions. More details on the use of these functions and their results can be found in the Introductory course. The use of sample_metadata is, for instance, explained in Workshop 1 Module 2 while more can be learned about CNVs in Workshop 2. We will also use some principal components analyses (PCA).

This module will also explain how to use unrestricted flags in conjunction with surveillance flags. Surveillance flags are introduced in Surveillance metadata flags, where more explanation on surveillance flags and how to use them can be found.

Learning objectives#

After completing this module, you will be able to:

Explain what it means for a sample set to be “unrestricted”
Understand when to use the unrestricted_use_only flags for your analyses
Use the metadata flags when needed

Installations and imports#

%pip install -q --no-warn-conflicts malariagen_data

import malariagen_data
import os

import plotly.io as pio
pio.renderers.default = "notebook+colab"

try:
    # if running on colab, mount Google Drive
    from google.colab import drive
    drive.mount('drive')
except ImportError:
    pass

results_dir = "drive/MyDrive/Colab Data/ag3-structure-results"
os.makedirs(results_dir, exist_ok=True)

What is an “unrestricted” sample set and why are some sample sets not “unrestricted”?#

Put simply, a sample set is “unrestricted” if anyone can use it however they so choose.

All the sample sets that are accessible using either Ag3 or Af1 are open but terms of use apply. These terms of use are available on the MalariaGEN website and may differ from sample set to sample set. For example, the terms of use for Anopheles gambiae Genomic Surveillance project (which can be found here) state that there is a “publication embargo [which] will expire 24 months after the data is integrated into the Malaria Genome Vector Observatory data repository, or earlier, if the project partner agrees to remove the embargo before the expiry date.”.

We created a flag, unrestricted_use, to facilitate checking whether a sample set is out of embargo. It is True if the sample set can be used without any restriction and False otherwise. Users can access and explore sample sets with a False unrestricted_use flag through the package but they need to be aware that terms of use apply and these differ between sample sets. It is key to respect these terms of use when accessing MalariaGEN resources, this allow us to continue working with the data producers/owners to generate key resources for the community.

Standard setup used during the entry level course#

Let us first look at the setup you might be more familiar with for Ag3. We could have used Af1 in the same way but we assume that users are, at this point, more familiar with Ag3.

ag3_default = malariagen_data.Ag3(results_cache=results_dir)

We will start by looking at how many sample sets are “unrestricted”.

ag3_default.sample_sets()['unrestricted_use'].value_counts()

unrestricted_use
True     83
False    18
Name: count, dtype: Int64

We see that more than half the sample sets are “unrestricted”. We can look at the first few “restricted” sample sets.

ag3_default.sample_sets().query('unrestricted_use == False').head()

/var/tmp/ipykernel_7518/3652335835.py:1: RuntimeWarning: Engine has switched to 'python' because numexpr does not support extension array dtypes. Please set your engine to python manually.
  ag3_default.sample_sets().query('unrestricted_use == False').head()

	sample_set	sample_count	study_id	study_url	terms_of_use_expiry_date	terms_of_use_url	release	unrestricted_use
33	1296-VO-BF-DIABATE-VMF00272	665	1296-VO-BF-DIABATE	https://www.malariagen.net/network/where-we-wo...	2026-09-06	https://malariagen.github.io/vector-data/ag3/a...	3.11	False
34	1351-VO-SS-WEETMAN-VMF00282	90	1351-VO-SS-WEETMAN	https://www.malariagen.net/network/where-we-wo...	2026-09-06	https://malariagen.github.io/vector-data/ag3/a...	3.11	False
35	1338-VO-NG-ADEDAPO-VMF00268	720	1338-VO-NG-ADEDAPO	https://www.malariagen.net/network/where-we-wo...	2026-11-04	https://malariagen.github.io/vector-data/ag3/a...	3.12	False
36	1324-VO-ET-GOLASSA-VMF00257	420	1324-VO-ET-GOLASSA	https://www.malariagen.net/network/where-we-wo...	2026-11-04	https://malariagen.github.io/vector-data/ag3/a...	3.13	False
37	1324-VO-ET-GOLASSA-VMF00275	573	1324-VO-ET-GOLASSA	https://www.malariagen.net/network/where-we-wo...	2027-03-13	https://malariagen.github.io/vector-data/ag3/a...	3.14	False

This cell was run on 20/08/2026. The sample set on the last row that we see is ‘1324-VO-ET-GOLASSA-VMF00275’. If we look to the terms_of_use_expiry_date column, we can see that it is under embargo until 13/03/2027. As it is later than the day this cell was run, it makes sense that is not yet unrestricted. You may get a very different output depending on when you run this cell.

Unrestricted setup#

Let us now set up things so that we only access the unrestricted sample sets.

# Construct an Ag3 object using the `unrestricted_use_only` setting.
ag3_unrestricted = malariagen_data.Ag3(unrestricted_use_only=True)

Let us check that only the unrestricted sample sets are part of this resource.

# See the value counts for `unrestricted_use` for all of the sample sets relevant to this object.
ag3_unrestricted.sample_sets()['unrestricted_use'].value_counts()

unrestricted_use
True    83
Name: count, dtype: Int64

We see that all the sample sets are “unrestricted”. We can try to look at the first few “restricted” sample sets.

ag3_unrestricted.sample_sets().query('unrestricted_use == False').head()

/var/tmp/ipykernel_7518/2673686400.py:1: RuntimeWarning: Engine has switched to 'python' because numexpr does not support extension array dtypes. Please set your engine to python manually.
  ag3_unrestricted.sample_sets().query('unrestricted_use == False').head()

	sample_set	sample_count	study_id	study_url	terms_of_use_expiry_date	terms_of_use_url	release	unrestricted_use

We can’t find any.

Combining unrestricted sample sets and surveillance samples#

Surveillance flags are introduced in Surveillance metadata flags, where more explanation on surveillance flags and how to use them can be found.

Working with unrestricted sample sets and surveillance samples at the same time is possible.

Standard setup used during the entry level course#

Let’s look only at the number of surveillance samples in each unrestricted sample set in Ag3.6. We will start by listing the unrestricted sample sets.

unrestricted_sample_sets = list(ag3_default.sample_sets(release='3.6')[ag3_default.sample_sets()['unrestricted_use']].sample_set)
unrestricted_sample_sets

/var/tmp/ipykernel_7518/1887060705.py:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  unrestricted_sample_sets = list(ag3_default.sample_sets(release='3.6')[ag3_default.sample_sets()['unrestricted_use']].sample_set)

['1273-VO-ZM-MULEBA-VMF00176',
 '1279-VO-CI-KOFFI-VMF00173',
 '1280-VO-ZA-MUNHENGA-VMF00165',
 '1280-VO-ZA-MUNHENGA-VMF00178',
 '1288-VO-UG-DONNELLY-VMF00168']

We can then ask for the number of surveillance samples in each of these sample sets.

ag3_default.sample_metadata(sample_sets=unrestricted_sample_sets).query('is_surveillance').groupby('sample_set').size()

/var/tmp/ipykernel_7518/138774416.py:1: RuntimeWarning: Engine has switched to 'python' because numexpr does not support extension array dtypes. Please set your engine to python manually.
  ag3_default.sample_metadata(sample_sets=unrestricted_sample_sets).query('is_surveillance').groupby('sample_set').size()

sample_set
1273-VO-ZM-MULEBA-VMF00176      201
1279-VO-CI-KOFFI-VMF00173       379
1280-VO-ZA-MUNHENGA-VMF00165    178
1280-VO-ZA-MUNHENGA-VMF00178    163
1288-VO-UG-DONNELLY-VMF00168    483
dtype: int64

Unrestricted setup#

This could be done easily by using the unrestricted setup.

ag3_unrestricted.sample_metadata(sample_sets='3.6').query('is_surveillance').groupby('sample_set').size()

/var/tmp/ipykernel_7518/159947851.py:1: RuntimeWarning: Engine has switched to 'python' because numexpr does not support extension array dtypes. Please set your engine to python manually.
  ag3_unrestricted.sample_metadata(sample_sets='3.6').query('is_surveillance').groupby('sample_set').size()

sample_set
1273-VO-ZM-MULEBA-VMF00176      201
1279-VO-CI-KOFFI-VMF00173       379
1280-VO-ZA-MUNHENGA-VMF00165    178
1280-VO-ZA-MUNHENGA-VMF00178    163
1288-VO-UG-DONNELLY-VMF00168    483
dtype: int64

Combined setup#

However, the easiest solution would be to combine both.

# Construct an Ag3 object using both the `unrestricted_use_only` setting and the `surveillance_use_only` setting.
ag3_unrestricted_surveillance = malariagen_data.Ag3(unrestricted_use_only=True, surveillance_use_only=True)

ag3_unrestricted_surveillance.sample_metadata(sample_sets='3.6').groupby('sample_set').size()

/opt/conda/envs/mgenv_7.3.0/lib/python3.12/site-packages/malariagen_data/anoph/sample_metadata.py:910: RuntimeWarning: Engine has switched to 'python' because numexpr does not support extension array dtypes. Please set your engine to python manually.
  df_samples = df_samples.query(prepared_sample_query, **sample_query_options)

sample_set
1273-VO-ZM-MULEBA-VMF00176      201
1279-VO-CI-KOFFI-VMF00173       379
1280-VO-ZA-MUNHENGA-VMF00165    178
1280-VO-ZA-MUNHENGA-VMF00178    163
1288-VO-UG-DONNELLY-VMF00168    483
dtype: int64

We can check that only unrestricted sample sets and surveillance samples are available.

# See the value counts for `unrestricted_use` for all of the sample sets relevant to this object.
# Note that only sample sets with `unrestricted_use` set to `True` are returned for this object.
ag3_unrestricted_surveillance.sample_sets()['unrestricted_use'].value_counts()

unrestricted_use
True    73
Name: count, dtype: Int64

ag3_unrestricted_surveillance.sample_metadata()['is_surveillance'].value_counts()

/opt/conda/envs/mgenv_7.3.0/lib/python3.12/site-packages/malariagen_data/anoph/sample_metadata.py:910: RuntimeWarning: Engine has switched to 'python' because numexpr does not support extension array dtypes. Please set your engine to python manually.
  df_samples = df_samples.query(prepared_sample_query, **sample_query_options)

is_surveillance
True    17291
Name: count, dtype: Int64

Which flags have been set can be seen by looking at the resource.

ag3_unrestricted_surveillance

MalariaGEN Ag3 API client
Please note that data are subject to terms of use, for more information see the MalariaGEN website or contact support@malariagen.net. See also the Ag3 API docs.
Storage URL	gs://vo_agam_release_master_us_central1
Data releases available	3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10, 3.11, 3.12, 3.13, 3.14, 3.15, 3.16
Results cache	None
Cohorts analysis	20260120
AIM analysis	20220528
Site filters analysis	dt_20200416
Software version	malariagen_data 15.8.0
Client location	Iowa, United States (Google Cloud us-central1)
Data filtered for unrestricted use only	True
Data filtered for surveillance use only	True
Relevant data releases	3.0, 3.1, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10

In addition to the sample_sets and the sample_metadata functions, other functions and properties in the package also return data differently depending on the setting of the unrestricted_use_only and surveillance_use_only parameters. As a general rule, all functions and properties that appear in the API documentation should honour these settings, but you should check first. Be aware that so-called “private” functions, which are used internally in the package and have an underscore prefix, e.g. _surveillance_flags, might return unfiltered data regardless of the settings.

Let us, for instance, look at the set of releases in each configuration of Ag3. release lists the releases with at least one sample set in the resource.

ag3_default.releases

('3.0',
 '3.1',
 '3.2',
 '3.3',
 '3.4',
 '3.5',
 '3.6',
 '3.7',
 '3.8',
 '3.9',
 '3.10',
 '3.11',
 '3.12',
 '3.13',
 '3.14',
 '3.15',
 '3.16')

ag3_unrestricted.releases

('3.0', '3.1', '3.2', '3.3', '3.4', '3.5', '3.6', '3.7', '3.8', '3.9', '3.10')

ag3_unrestricted_surveillance.releases

('3.0', '3.1', '3.3', '3.4', '3.5', '3.6', '3.7', '3.8', '3.9', '3.10')

ag3_default contains all the currently available releases while ag3_unrestricted and ag3_unrestricted_surveillance contain some subsets of the available releases.

Depending on which flags are set, the number of samples returned during an analysis may vary greatly. Let us look at an example showing the CNV HMMs for ‘1274-VO-KE-KAMAU-VMF00246’ with default configuration and the unrestricted and surveillance one.

# The `aim_calls` function will return samples depending on the object's `surveillance_use_only` setting. For example:
cnv_region = '2R:28,480,000-28,490,000'
ag3_cnv_hmm_df = ag3_default.cnv_hmm(region=cnv_region, sample_sets='1274-VO-KE-KAMAU-VMF00246').to_dataframe()
ag3_unrestricted_surveillance_cnv_hmm_df = ag3_unrestricted_surveillance.cnv_hmm(region=cnv_region, sample_sets='1274-VO-KE-KAMAU-VMF00246').to_dataframe()
print('Samples returned by the default object:', len(ag3_cnv_hmm_df['sample_id'].unique()))
print('Samples returned by the unrestricted and surveillance object:', len(ag3_unrestricted_surveillance_cnv_hmm_df['sample_id'].unique()))

Access CNV HMM data: ⠧ (0:00:00.58)

/opt/conda/envs/mgenv_7.3.0/lib/python3.12/site-packages/malariagen_data/anoph/sample_metadata.py:910: RuntimeWarning: Engine has switched to 'python' because numexpr does not support extension array dtypes. Please set your engine to python manually.
  df_samples = df_samples.query(prepared_sample_query, **sample_query_options)
/opt/conda/envs/mgenv_7.3.0/lib/python3.12/site-packages/malariagen_data/anoph/base.py:987: RuntimeWarning: Engine has switched to 'python' because numexpr does not support extension array dtypes. Please set your engine to python manually.
  loc_samples = df_samples.eval(sample_query, **sample_query_options)

Samples returned by the default object: 359
Samples returned by the unrestricted and surveillance object: 49

We get very different numbers of samples and, if we display the results, we will also get different plots.

ag3_default.plot_cnv_hmm_heatmap(
    region=cnv_region, 
    sample_sets='1274-VO-KE-KAMAU-VMF00246',
    row_height=5
);

ag3_unrestricted_surveillance.plot_cnv_hmm_heatmap(
    region=cnv_region, 
    sample_sets='1274-VO-KE-KAMAU-VMF00246',
    row_height=5
);

/opt/conda/envs/mgenv_7.3.0/lib/python3.12/site-packages/malariagen_data/anoph/sample_metadata.py:910: RuntimeWarning: Engine has switched to 'python' because numexpr does not support extension array dtypes. Please set your engine to python manually.
  df_samples = df_samples.query(prepared_sample_query, **sample_query_options)
/opt/conda/envs/mgenv_7.3.0/lib/python3.12/site-packages/malariagen_data/anoph/base.py:987: RuntimeWarning: Engine has switched to 'python' because numexpr does not support extension array dtypes. Please set your engine to python manually.
  loc_samples = df_samples.eval(sample_query, **sample_query_options)

The general distribution of results is broadly similar between the two plots. One can observe roughly 3 different patterns: no amplification, an amplification that covers the whole region and an amplification that covers only Cyp6aa1. However, the number of samples, and thus the number of rows, is completely different.

Well done

In this module, we have learnt:

Why the use of some data may be restricted
How to configure the resources to only access the data that is relevant to our scope
That one might get different results with different configurations. One needs to be careful.

Congratulations on reaching the end of this notebook.

Unrestricted metadata flags

Contents

Unrestricted metadata flags#

Learning objectives#

Installations and imports#

What is an “unrestricted” sample set and why are some sample sets not “unrestricted”?#

Standard setup used during the entry level course#

Unrestricted setup#

Combining unrestricted sample sets and surveillance samples#

Standard setup used during the entry level course#

Unrestricted setup#

Combined setup#