Loading and storing IMAS data¶

IMAS data is grouped together in Data Entries. A Data Entry is a collection of IDSs and their (potentially) multiple occurrences, which groups and stores data over multiple IDSs as a single dataset. The Data Entry concept is used whether the collection of IDSs is stored in a database or only exists temporarily (for example for communication in an integrated workflow).

Loading and storing IMAS data happens through an IMAS Database Entry. A Database Entry tracks the information required for locating where the Data Entry is (or will be) stored on disk. In Fortran this object is modeled as an integer index.

You may Open an existing IMAS Database Entry, which you can use for loading data that was stored previously. Alternatively you can Create a new IMAS Database Entry to store IDS data.

Open an existing IMAS Database Entry¶

To open an IMAS Database Entry, you need to know the URI indicating where the Access Layer can find the data. IMAS URIs start with imas: and indicate the format and the location of the stored data. You can find a detailed description of the IMAS URI syntax on the IMAS Data Entry URIs page.

program open_data_entry
    ! Make the Access Layer available
    use ids_routines
    implicit none

    integer :: data_entry, data_entry2, status

    ! Open the database entry by providing an IMAS URI
    call imas_open( &
        "imas:mdsplus?user=public;pulse=131024;run=41;database=ITER;version=3", &
        OPEN_PULSE, data_entry, status)

    ! Alternatively, we can open an IDS providing legacy arguments
    call imas_open_env("ids", 131024, 41, data_entry2, "public", "ITER", "3", status)

end program open_data_entry

Loading IMAS data¶

After you open a database entry, you can request to load data from disk.

Load an entire IDS¶

With ids_get() you can load (“get”) an entire IDS from the database entry.

Multiple occurrences of an IDS may be stored in a data entry. By default, if you don’t specify an occurrence number, occurrence 0 is loaded. By providing an occurrence number you can load a specific occurrence. How different occurrences are used depends on the experiment. They could, for example, correspond to:

• different methods for computing the physical quantities of the IDS, or

• different functionalities in a workflow (e.g. initial values, prescribed values, values at next time step, …), or

• multiple subsystems (e.g. diagnostics) of the same type in an experiment, etc.

program load_ids
    ! Make the Access Layer available
    use ids_routines
    implicit none

    integer :: data_entry, status
    type(ids_core_profiles) :: cp, cp1

    ! Open the database entry by providing an IMAS URI
    call imas_open( &
        "imas:mdsplus?user=public;pulse=131024;run=41;database=ITER;version=3", &
        OPEN_PULSE, data_entry, status)

    ! Load the core_profiles IDS, occurrence 0
    call ids_get(data_entry, "core_profiles", cp)

    ! Load the core_profiles IDS, occurrence 1
    call ids_get(data_entry, "core_profiles/1", cp1)

end program load_ids

Load a single time slice of an IDS¶

Instead of loading a full IDS from disk, the Access Layer allows you to load a specific time slice. This is often useful when you’re not interested in the full time evolution, but instead want data of a specific time. You can use ids_get_slice() for this.

Most of the time there are no entries at that specific time, so you also need to indicate an interpolation method. This determines what values the access layer returns when your requested time is in between available time points in the data. Three interpolation methods currently exist:

PREVIOUS_INTERP

Returns the previous time slice if the requested time does not exactly exist in the original IDS.

For example, when data exists at \(t=\{1, 3, 4\}\), requesting \(t_r=2.1\) will give you the data at \(t=1\).

Edge case behaviour. \(\{t_i\}, i=1..N\) represents the time series stored in the IDS.¶

Case	Behaviour
\(t_r \lt t_1\)	Return data at \(t_1\).
\(t_r = t_i\) [1]	Return data at \(t_i\).

CLOSEST_INTERP

Returns the closest time slice in the original IDS. This can also be after the requested time.

For example, when data exists at \(t=\{1, 3, 4\}\), requesting \(t=2.1\) will give you the data at \(t=3\).

Edge case behaviour. \(\{t_i\}, i=1..N\) represents the time series stored in the IDS.¶

Case	Behaviour
\(t_r \lt t_1\)	Return data at \(t_1\).
\(t_r = t_i\) [1]	Return data at \(t_i\).
\(t_r - t_i = t_{i+1} - t_r\) [1]	Return data at \(t_{i+1}\).

LINEAR_INTERP

Returns a linear interpolation between the existing slices before and after the requested time.

For example, when data exists at \(t=\{1, 3, 4\}\), requesting \(t=2.1\) will give you a linear interpolation of the data at \(t=1\) and the data at \(t=3\).

Note that the linear interpolation will be successful only if, between the two time slices of an interpolated dynamic array of structure, the same leaves are populated and they have the same size. Otherwise ids_get_slice() will interpolate all fields with a compatible size and leave others empty.

Edge case behaviour. \(\{t_i\}, i=1..N\) represents the time series stored in the IDS.¶

Case	Behaviour
\(t_r \lt t_1\)	Return data at \(t_1\).
\(t_r \gt t_N\)	Return data at \(t_N\).

program load_ids_slice
    ! Make the Access Layer available
    use ids_routines
    implicit none

    integer(ids_int) :: data_entry, status
    real(ids_real) :: twant
    type(ids_core_profiles) :: cp

    ! Open the database entry by providing an IMAS URI
    call imas_open( &
        "imas:mdsplus?user=public;pulse=131024;run=41;database=ITER;version=3", &
        OPEN_PULSE, data_entry, status)

    ! Retrieve the time slice just before t=370 of the core_profiles IDS
    twant = 370
    call ids_get_slice(data_entry, "core_profiles", cp, twant, PREVIOUS_INTERP)

end program load_ids_slice

Create a new IMAS Database Entry¶

To create a new IMAS Database Entry, you need to provide the URI to indicate the format and the location where you want to store the data. You can find a detailed description of the IMAS URI syntax and the options available on the IMAS Data Entry URIs page.

program create_data_entry
    ! Make the Access Layer available
    use ids_routines
    implicit none

    integer :: data_entry, status, pulse, run
    character(len=132) :: user

    ! Create the database entry by providing an IMAS URI
    call imas_open("imas:hdf5?path=my-data", FORCE_CREATE_PULSE, data_entry, status)
    
    ! Alternatively, we can open an IDS providing legacy arguments
    pulse = 1
    run = 1
    call get_environment_variable("USER", user)
    call imas_create_env("ids", pulse, run, 0, 0, data_entry, user, "TEST", "3", status)

end program create_data_entry

Store IMAS data¶

After you have created an IMAS Database Entry, you can use it for storing IDS data. There are two ways to do this:

Store an entire IDS¶

With ids_put() you can store (“put”) an entire IDS in a database entry. First you need to have an IDS with data: you can create a new one or load an IDS which you modify. See Use Interface Data Structures for more information on using and manipulating IDSs.

Multiple occurrences of an IDS may be stored in a data entry. By default, if you don’t specify an occurrence number, the IDS is stored as occurrence 0. By providing an occurrence number you can store the IDS as a specific occurrence.

program store_ids
    ! Make the Access Layer available
    use ids_routines
    implicit none

    integer :: data_entry, status
    type(ids_pf_active) :: pf_active

    ! Create the database entry by providing an IMAS URI
    call imas_open("imas:hdf5?path=my-data", FORCE_CREATE_PULSE, data_entry, status)

    ! Set the mandatory ids_properties.homogeneous_time field
    pf_active%ids_properties%homogeneous_time = IDS_TIME_MODE_HOMOGENEOUS
    ! Continue filling the pf_active IDS here
    ! ...

    ! Store the pf_active IDS
    call ids_put(data_entry, "pf_active", pf_active)

    ! Alternatively, store the pf_active IDS as occurrence 1
    call ids_put(data_entry, "pf_active/1", pf_active)

end program store_ids

Append a time slice to an already-stored IDS¶

With ids_put_slice() you can append a time slice to an existing database entry. This is useful when you generate data inside a time loop (for example in simulations, or when taking measurements of an experiment).

It means you can put a time slice with every iteration of your loop such that you don’t have to keep track of the complete time evolution in memory. Instead, the Access Layer will keep appending the data to the Database Entry in the storage backend.

program store_ids_slice
    ! Make the Access Layer available
    use ids_routines
    implicit none

    integer :: data_entry, status
    real(ids_real) :: t, t_stop, dt
    type(ids_core_profiles) :: core_profiles

    ! Create the database entry by providing an IMAS URI
    call imas_open("imas:hdf5?path=my-data", FORCE_CREATE_PULSE, data_entry, status)

    ! Set the mandatory ids_properties.homogeneous_time field
    core_profiles%ids_properties%homogeneous_time = IDS_TIME_MODE_HOMOGENEOUS
    ! Continue filling the core_profiles IDS with static values here
    ! ...

    ! Allocate time array
    allocate(core_profiles%time(1))

    ! Example time integration loop:
    t = 0.0
    t_stop = 60.0
    dt = 0.1

    do while (t .lt. t_stop)
        core_profiles%time = (/ t /)
        ! Fill your time-dependent data in the core_profiles IDS here
        ! ...

        ! Append the current time slice to the data stored on disk
        call ids_put_slice(data_entry, "core_profiles", core_profiles)

        t = t + dt
    end do

end program store_ids_slice

Listing all occurrences of an IDS from a backend¶

With list_all_occurrences() you can List all non-empty occurrences of an IDS using its name in the dataset, and optionnally return the content of a descriptive node path.

program prog_list_all_occurrences
    ! Make the Access Layer available
    use ids_routines
    implicit none

    integer :: data_entry, status
    character(len=:), allocatable :: node_content_list(:)
    integer, allocatable          :: occurrence_list(:)
    ! Open the database entry by providing an IMAS URI
    call imas_open( &
        "imas:mdsplus?user=public;pulse=131024;run=41;database=ITER;version=3", &
        OPEN_PULSE, data_entry, status)

    call list_all_occurrences(data_entry, "core_profiles", "ids_properties/comment", node_content_list, occurrence_list)

    if(allocated(occurrence_list)) then 
        write(*,*) "i     occurrence_list(i)      node_content_list(i)"
            do i = 1, size(occurrence_list)
                write(*,*) i,occurrence_list(i),"'"//node_content_list(i)//"'"
            end do
    else 
        write(*,*) "node_content_list, occurrence_list are empty"
    end if 

end program prog_list_all_occurrences