Loading Part 4: Settings Files¶

In this part of the tutorial, we will explore how to save and load stream and lake settings dictionaries. This can be incredibly useful for preserving your settings, sharing them with others, or simply avoiding the need to retype everything.

Working Path¶

In your working path, you will find a couple of .json files. These files are the settings files. The lake_settings.json file stores the settings for the lake, while the stream_settings.json file stores the settings for the stream. These settings are the same ones you created in the previous example, but now they are saved to files for easy access and sharing.

data
├── CPC_3010_data
│   ├── CPC_3010_data_20220709_Jul.csv
│   ├── CPC_3010_data_20220709_Jul.csv
    ├── stream_settings_cpc.json
├── SMPS_data
│   ├── 2022-07-07_095151_SMPS.csv
│   ├── 2022-07-10_094659_SMPS.csv
│   ├── stream_settings_smps_1d.json
│   ├── stream_settings_smps_2d.json
├── lake_settings.json

In [15]:

# Import the necessary libraries and modules
import matplotlib.pyplot as plt
from particula_beta.data import loader_interface, settings_generator
from particula_beta.data.tests.example_data.get_example_data import get_data_folder
from particula_beta.data.lake import Lake
from particula_beta.data.loader_setting_builders import (
    # These functions create settings for loading data from files.
    DataChecksBuilder,
    SizerDataReaderBuilder,
    Loader1DSettingsBuilder,
    LoaderSizerSettingsBuilder,
)

# Set the parent directory where the data folders are located
path = get_data_folder()
print('Path to data folder:')
print(path.rsplit('particula')[-1])

# Import the necessary libraries and modules import matplotlib.pyplot as plt from particula_beta.data import loader_interface, settings_generator from particula_beta.data.tests.example_data.get_example_data import get_data_folder from particula_beta.data.lake import Lake from particula_beta.data.loader_setting_builders import ( # These functions create settings for loading data from files. DataChecksBuilder, SizerDataReaderBuilder, Loader1DSettingsBuilder, LoaderSizerSettingsBuilder, ) # Set the parent directory where the data folders are located path = get_data_folder() print('Path to data folder:') print(path.rsplit('particula')[-1])

Path to data folder:
\data\tests\example_data

Generate and Save Settings¶

First, we generate the settings for the CPC data using the settings_generator.for_general_1d_load function. These settings include details such as the data file location, file format, column names, and more.

In [16]:

# settings for the CPC data
data_checks_cpc = (
    DataChecksBuilder()
    .set_characters([10, 100])
    .set_char_counts({",": 4})
    .build()
)

cpc_settings = (
    Loader1DSettingsBuilder()
    .set_relative_data_folder("CPC_3010_data")
    .set_filename_regex("*.csv")
    .set_header_row(0)
    .set_data_checks(data_checks_cpc)  # from above
    .set_data_column([1, 2])
    .set_data_header(["CPC_count[#/sec]", "Temperature[degC]"])
    .set_time_column([0])
    .set_time_format("epoch")
    .set_delimiter(",")
    .set_timezone_identifier("UTC")
    .build()
)

# save the settings to a file
settings_generator.save_settings_for_stream(
    settings=cpc_settings,
    path=path,
    subfolder='CPC_3010_data',
    settings_suffix='_cpc',
)

# settings for the CPC data data_checks_cpc = ( DataChecksBuilder() .set_characters([10, 100]) .set_char_counts({",": 4}) .build() ) cpc_settings = ( Loader1DSettingsBuilder() .set_relative_data_folder("CPC_3010_data") .set_filename_regex("*.csv") .set_header_row(0) .set_data_checks(data_checks_cpc) # from above .set_data_column([1, 2]) .set_data_header(["CPC_count[#/sec]", "Temperature[degC]"]) .set_time_column([0]) .set_time_format("epoch") .set_delimiter(",") .set_timezone_identifier("UTC") .build() ) # save the settings to a file settings_generator.save_settings_for_stream( settings=cpc_settings, path=path, subfolder='CPC_3010_data', settings_suffix='_cpc', )

Next save the SMPS settings¶

In [17]:

# settings for the SMPS data
data_checks_sizer = (
    DataChecksBuilder()
    .set_characters([250])
    .set_skip_rows(25)
    .set_char_counts({"/": 2, ":": 2})
    .build()
)
data_sizer_reader = (
    SizerDataReaderBuilder()
    .set_sizer_start_keyword("20.72")
    .set_sizer_end_keyword("784.39")
    .set_sizer_concentration_convert_from("dw/dlogdp")
    .build()
)
smps_1d_settings, smps_2d_settings = (
    LoaderSizerSettingsBuilder()
    .set_relative_data_folder("SMPS_data")
    .set_filename_regex("*.csv")
    .set_header_row(24)
    .set_data_checks(data_checks_sizer)
    .set_data_column(
        [
            "Lower Size (nm)",
            "Upper Size (nm)",
            "Sample Temp (C)",
            "Sample Pressure (kPa)",
            "Relative Humidity (%)",
            "Median (nm)",
            "Mean (nm)",
            "Geo. Mean (nm)",
            "Mode (nm)",
            "Geo. Std. Dev.",
            "Total Conc. (#/cm³)",
        ]
    )
    .set_data_header(
        [
            "Lower_Size_(nm)",
            "Upper_Size_(nm)",
            "Sample_Temp_(C)",
            "Sample_Pressure_(kPa)",
            "Relative_Humidity_(%)",
            "Median_(nm)",
            "Mean_(nm)",
            "Geo_Mean_(nm)",
            "Mode_(nm)",
            "Geo_Std_Dev.",
            "Total_Conc_(#/cc)",
        ]
    )
    .set_data_sizer_reader(data_sizer_reader)
    .set_time_column([1, 2])
    .set_time_format("%m/%d/%Y %H:%M:%S")
    .set_delimiter(",")
    .set_timezone_identifier("UTC")
    .build()
)

# save the settings to a file
settings_generator.save_settings_for_stream(
    settings=smps_1d_settings,
    path=path,
    subfolder='SMPS_data',
    settings_suffix='_smps_1d',
)
settings_generator.save_settings_for_stream(
    settings=smps_2d_settings,
    path=path,
    subfolder='SMPS_data',
    settings_suffix='_smps_2d',
)

# settings for the SMPS data data_checks_sizer = ( DataChecksBuilder() .set_characters([250]) .set_skip_rows(25) .set_char_counts({"/": 2, ":": 2}) .build() ) data_sizer_reader = ( SizerDataReaderBuilder() .set_sizer_start_keyword("20.72") .set_sizer_end_keyword("784.39") .set_sizer_concentration_convert_from("dw/dlogdp") .build() ) smps_1d_settings, smps_2d_settings = ( LoaderSizerSettingsBuilder() .set_relative_data_folder("SMPS_data") .set_filename_regex("*.csv") .set_header_row(24) .set_data_checks(data_checks_sizer) .set_data_column( [ "Lower Size (nm)", "Upper Size (nm)", "Sample Temp (C)", "Sample Pressure (kPa)", "Relative Humidity (%)", "Median (nm)", "Mean (nm)", "Geo. Mean (nm)", "Mode (nm)", "Geo. Std. Dev.", "Total Conc. (#/cm³)", ] ) .set_data_header( [ "Lower_Size_(nm)", "Upper_Size_(nm)", "Sample_Temp_(C)", "Sample_Pressure_(kPa)", "Relative_Humidity_(%)", "Median_(nm)", "Mean_(nm)", "Geo_Mean_(nm)", "Mode_(nm)", "Geo_Std_Dev.", "Total_Conc_(#/cc)", ] ) .set_data_sizer_reader(data_sizer_reader) .set_time_column([1, 2]) .set_time_format("%m/%d/%Y %H:%M:%S") .set_delimiter(",") .set_timezone_identifier("UTC") .build() ) # save the settings to a file settings_generator.save_settings_for_stream( settings=smps_1d_settings, path=path, subfolder='SMPS_data', settings_suffix='_smps_1d', ) settings_generator.save_settings_for_stream( settings=smps_2d_settings, path=path, subfolder='SMPS_data', settings_suffix='_smps_2d', )

Loading Stream Settings¶

If you are still exploring your analysis pipeline, you may want to load settings for individual streams. To do so, you can use the generate_settings.load_settings_for_stream function. This function takes the path to the settings file as an argument and returns a dictionary containing the stream settings.

In [18]:

smps_1d_stream_settings = settings_generator.load_settings_for_stream(
    path=path,
    subfolder='SMPS_data',
    settings_suffix='_smps_1d',
)

stream_smps_1d = loader_interface.load_files_interface(
    path=path,
    settings=smps_1d_stream_settings
)

print(stream_smps_1d.header)

smps_1d_stream_settings = settings_generator.load_settings_for_stream( path=path, subfolder='SMPS_data', settings_suffix='_smps_1d', ) stream_smps_1d = loader_interface.load_files_interface( path=path, settings=smps_1d_stream_settings ) print(stream_smps_1d.header)

  Loading file: 2022-07-07_095151_SMPS.csv
  Loading file: 2022-07-10_094659_SMPS.csv
['Lower_Size_(nm)', 'Upper_Size_(nm)', 'Sample_Temp_(C)', 'Sample_Pressure_(kPa)', 'Relative_Humidity_(%)', 'Median_(nm)', 'Mean_(nm)', 'Geo_Mean_(nm)', 'Mode_(nm)', 'Geo_Std_Dev.', 'Total_Conc_(#/cc)']

Lake settings¶

If you wanted to load everything for a reanalysis, instead of calling each individual stream, you can first save a lake settings file. This is done with generate_settings.save_settings_for_lake. This function takes the path to the lake settings file as an argument. It returns a dictionary with the settings.

In [19]:

# collect settings into a dictionary
combined_settings = {
    'cpc': cpc_settings,
    'smps_1d': smps_1d_settings,
    'smps_2d': smps_2d_settings,
}

# save the lake settings to a file
settings_generator.save_settings_for_lake(
    settings=combined_settings,
    path=path,
    subfolder='',
    settings_suffix='_cpc_smps',
)

# collect settings into a dictionary combined_settings = { 'cpc': cpc_settings, 'smps_1d': smps_1d_settings, 'smps_2d': smps_2d_settings, } # save the lake settings to a file settings_generator.save_settings_for_lake( settings=combined_settings, path=path, subfolder='', settings_suffix='_cpc_smps', )

Load the Lake¶

To load the lake settings use generate_settings.load_settings_for_lake. This function takes the path to the lake settings file as an argument. It returns a dictionary with the settings.

In [20]:

lake_settings = settings_generator.load_settings_for_lake(
    path=path,
    subfolder='',
    settings_suffix='_cpc_smps',
)

# now call the loader interface for files
lake = loader_interface.load_folders_interface(
    path=path,
    folder_settings=combined_settings,
)

print(' ')
print(lake)

lake_settings = settings_generator.load_settings_for_lake( path=path, subfolder='', settings_suffix='_cpc_smps', ) # now call the loader interface for files lake = loader_interface.load_folders_interface( path=path, folder_settings=combined_settings, ) print(' ') print(lake)

Folder Settings: cpc
  Loading file: CPC_3010_data_20220709_Jul.csv
  Loading file: CPC_3010_data_20220710_Jul.csv
Folder Settings: smps_1d
  Loading file: 2022-07-07_095151_SMPS.csv
  Loading file: 2022-07-10_094659_SMPS.csv
Folder Settings: smps_2d
  Loading file: 2022-07-07_095151_SMPS.csv
  Loading file: 2022-07-10_094659_SMPS.csv
 
Lake with streams: ['cpc', 'smps_1d', 'smps_2d']

Summary¶

This example showed how to save and load the settings for a stream and a lake. This is useful if you want to save your settings and then load them later. This is also useful if you want to share your settings with someone else. Or just stop from having to retype everything.