diff --git a/documentation/.vuepress/components/LinkableChoices.vue b/documentation/.vuepress/components/LinkableChoices.vue index 95d2804..e6cca58 100644 --- a/documentation/.vuepress/components/LinkableChoices.vue +++ b/documentation/.vuepress/components/LinkableChoices.vue @@ -93,6 +93,11 @@ export default { } } + h3 { + font-size: 1rem; + text-align: center; + } + img { width: 35px; height: 35px; diff --git a/documentation/.vuepress/public/install/screen.svg b/documentation/.vuepress/public/install/screen.svg new file mode 100644 index 0000000..e50e9e6 --- /dev/null +++ b/documentation/.vuepress/public/install/screen.svg @@ -0,0 +1,22 @@ + + + + + + + + + + + \ No newline at end of file diff --git a/documentation/.vuepress/public/install/template.svg b/documentation/.vuepress/public/install/template.svg new file mode 100644 index 0000000..d11bc4d --- /dev/null +++ b/documentation/.vuepress/public/install/template.svg @@ -0,0 +1,26 @@ + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/documentation/.vuepress/public/install/world.svg b/documentation/.vuepress/public/install/world.svg new file mode 100644 index 0000000..d22c69e --- /dev/null +++ b/documentation/.vuepress/public/install/world.svg @@ -0,0 +1,4 @@ + + + + \ No newline at end of file diff --git a/documentation/dataInsertion/README.md b/documentation/dataInsertion/README.md index 7e6eb7b..425e683 100644 --- a/documentation/dataInsertion/README.md +++ b/documentation/dataInsertion/README.md @@ -1,51 +1,95 @@ -# Inserting data into DebiAI +# Inserting Data into DebiAI -Being a data visualization application, providing the project data to DebiAI is a required step. +As a data visualization application, providing project data to DebiAI is a required step. ## Requirements -### A DebiAI instance +### A Running DebiAI Instance -You will need to have a running DebiAI instance to insert you project data to. (see [Installation](../introduction/gettingStarted/installation/README.md)) +You need a running DebiAI instance to insert your project data. (See [Installation](../introduction/gettingStarted/installation/README.md)) -### Data +### Data Format Requirements -The data you want to analyze with DebiAI will need to respect a specific format. +The data you want to analyze in DebiAI must follow a specific format. -- **CSV like format** +- **CSV-like Format** - If your data can be represented in an array like format, adding them to DebiAI will be easy. The data can also support different levels of nesting (see [unfolding columns](../dashboard/unfolding/)). + If your data is structured in an array-like format, adding it to DebiAI is straightforward. DebiAI also supports different levels of nesting (see [Unfolding Columns](../dashboard/unfolding/)). -- **Data types** +- **Supported Data Types** DebiAI supports the following data types: - `num`: numerical values - `str`: string values - `bool`: boolean values - - `array`: array of values (see [unfolding columns](../dashboard/unfolding/)) - - `dict`: dictionary of values (see [unfolding columns](../dashboard/unfolding/)) + - `array`: arrays of values (see [Unfolding Columns](../dashboard/unfolding/)) + - `dict`: dictionary objects (see [Unfolding Columns](../dashboard/unfolding/)) + - `None`: missing values - Dates are supported by DebiAI, you can provide them as strings. + Dates are supported and should be provided as strings. -- **Missing values** +- **Handling Missing Values** - DebiAI supports data with missing values (`None`, `NaN` or `null` values) since 0.29.0. The missing values will be displayed as `null` by widgets that support them. Statistics about missing values will be displayed in the dashboard. + Since version 0.29.0, DebiAI supports missing values (`None`, `NaN`, or `null`). Widgets that support missing values will display them as `null`, and statistics about missing data will be available in the dashboard. -- **Samples size** +- **Sample Size Limitations** - It is not recommended to provide more than 2.000.000 samples, as it will take a long time to process. We are working on improving this limit. + Providing more than **2,000,000 samples** is not recommended, as it may significantly increase processing time. We are actively working on improving this limitation. -## There is currently two ways to insert data into DebiAI: +## Methods for Inserting Data into DebiAI -- ### [Python module](pythonModule/README.md#python-module) +There are currently two ways to insert data into DebiAI: -The main way to add provide the project data to the application is through the DebiAI Python module. -The module was designed to be used directly in your Python workflow, to add model results directly after its evaluation for example. +DebiAI architecture -- ### [Data providers](dataProviders/README.md#data-providers) + -A DebiAI data provider is a REST service that will expose your project to DebiAI. -DebiAI will directly ask for the data from your project making the data loading process very quick and customizable. Unlike the DebiAI Python module, the provided data won't have to be duplicated in the DebiAI application. +### **1. [Data Providers](dataProviders/README.md#data-providers) (Recommended)** -Making a data provider is the most efficient way to make your project data accessible to DebiAI, no matter the data base that your project is using. +A **DebiAI Data Provider** is a service that exposes your project data to DebiAI. This method allows DebiAI to directly retrieve metadata from your project, making data loading **fast** and **customizable**. + +✅ **Key benefits**: + +- No need to upload or duplicate data in DebiAI. +- Always up to date with the latest project data. +- Works with any files or databases used by your project. + +⚠️ **Limitations**: + +- Requires a custom implementation to expose your data. + +To simplify implementation, you can use the [DebiAI Data Provider Python module](https://github.com/debiai/easy-data-provider). + +### **2. [Python Module](pythonModule/README.md#python-module)** + +You can also insert data directly from your Python workflow using the [DebiAI Python module](https://github.com/debiai/py-debiai). This is useful for integrating new data or model results immediately after generation. + +✅ **Key benefits**: + +- Easier to implement. + +⚠️ **Limitations**: + +- Requires data duplication in DebiAI, increasing load time. +- Data updates must be done manually. + +While easier to implement, this method is less efficient than using a Data Provider. + +--- + +By following the recommended **Data Provider** approach, you ensure an optimized project data integration with DebiAI. diff --git a/documentation/dataInsertion/dataProviders/README.md b/documentation/dataInsertion/dataProviders/README.md index 9781480..d852ade 100644 --- a/documentation/dataInsertion/dataProviders/README.md +++ b/documentation/dataInsertion/dataProviders/README.md @@ -1,50 +1,50 @@ -# Data providers +# Data Providers -Making a data provider is the most efficient way to make your project data accessible to DebiAI. +Creating a **Data Provider** is the most efficient way to make your project data accessible to DebiAI. -A data provider is a service that you create that can respond to the data requests of DebiAI. This service can be made in **any language**, can use **any kind of databases** and can be hosted on **any platform** as long at the DebiAI data-provider's API is respected. So unlike the [Debiai Python module](../pythonModule/README.md), your project data won't be duplicated in DebiAI and **DebiAI will always analyze the latest data**. +A **Data Provider** is a service that responds to DebiAI's data requests. It can be implemented in **any language**, use **any database**, and be hosted on **any platform**, as long as it follows the **DebiAI Data Provider API**. -### How does it work? +Unlike the [DebiAI Python module](../pythonModule/README.md), this method **does not duplicate data** in DebiAI, ensuring that DebiAI always analyzes the latest version of your data. -DebiAI will ask your data provider to return the data that it needs to display the dashboard: +## How It Works -- Available Project lists -- Available data IDs -- Project data -- Available models and results -- Data selections (optional) +DebiAI queries your Data Provider to retrieve information for the dashboard: -DebiAI will also be able to send the data selections made by the user. +- **Project lists:** available projects +- **Data IDs:** available samples +- **Project data:** actual data used for analysis +- **Model results:** available models and outputs (optional) +- **Data selections:** user-defined data selections (optional) -### Pros and cons +Additionally, DebiAI can send data selections made by the user back to the provider: -- **Pros**: - - DebiAI will always analyze the latest data - - Your data will not be duplicated in DebiAI - - You can use any languages and databases - - You can host your data provider on any platform - - Better for long term projects -- **Cons**: - - You need to create a data provider (you can start with our [data provider templates](./quickStart.md#creation-of-a-data-provider)) - - You need to respect the DebiAI data-provider's API (we made it as simple as possible) +- **Project deletion** +- **Model deletion** +- **Selection creation and deletion** -### The API +## Pros & Cons -The Data-providers API as been described with OpenAPI 3.0. +✅ **Pros**: -- [Data-providers API Swagger documentation](https://petstore.swagger.io/?url=https://raw.githubusercontent.com/debiai/data-provider-nodejs-template/main/data-provider-API.yaml) -- [Data-providers API yaml file](https://github.com/debiai/data-provider-nodejs-template/blob/main/data-provider-API.yaml). +- **Always up to date** – DebiAI always analyzes the latest data. +- **No data duplication** – Saves storage space. +- **Flexibility** – Works with any programming language and database. +- **Platform-independent** – Can be hosted anywhere. +- **Ideal for middle to long-term projects**. -### Speed +⚠️ **Cons**: -The speed at which your data loads into DebiAI depends on how quickly your data provider can provide them. So it depends on the size of the data and the speed of your database. +- Requires an initial **custom implementation**, but it's a one-time setup. To simplify implementation, you can use the [DebiAI Data Provider Python module](https://github.com/debiai/easy-data-provider). -The quicker your data provider is, the quicker your data will be available in DebiAI. +## Performance Considerations -### Getting started +The **speed of data loading** into DebiAI depends on how quickly your Data Provider responds. This is influenced by: -To create your first data provider, read our [Quick start](quickStart/README.md). +- **Data size** – Larger datasets take longer to load. +- **Database performance** – A fast database speeds up response times. -::: warning Limitations -- The interface between data-providers and DebiAI is not yet stable, so the API is likely to change in the future. -::: \ No newline at end of file +Optimizing your Data Provider ensures **faster** data retrieval in DebiAI. + +## Getting Started + +To create your first Data Provider, check out our [Quick Start Guide](quickStart/README.md). diff --git a/documentation/dataInsertion/dataProviders/quickStart.md b/documentation/dataInsertion/dataProviders/quickStart.md index 9170d85..7f7980f 100644 --- a/documentation/dataInsertion/dataProviders/quickStart.md +++ b/documentation/dataInsertion/dataProviders/quickStart.md @@ -1,81 +1,127 @@ -# Quick start - -## Creation of a data provider - -To help you with the creation of data-providers, we have created some quick start templates. At the moment the templates are available for the following languages: - -- Node.js : [https://github.com/debiai/data-provider-nodejs-template](https://github.com/debiai/data-provider-nodejs-template) - -If you think that we should add a template for another language, [please let us know](https://github.com/debiai/data-provider-nodejs-template/issues/new). - -::: tip Starting from scratch -You can create your own data-provider as long as the [API](https://petstore.swagger.io/?url=https://raw.githubusercontent.com/debiai/data-provider-nodejs-template/main/data-provider-API.yaml) is respected. -::: - -## Giving DebiAI access to a data provider - -Once you have created and deployed your data provider, you need to tell DebiAI where to find it. - -You can do this in three ways: - -- [From the dashboard](#from-the-dashboard) -- [With the environment variables (recommended)](#environment-variables) -- [With the configuration file](#configuration-file) - -::: warning -DebiAI needs to be able to access your data provider. - -If your data provider is deployed on the same machine as DebiAI, you can use `localhost` as the URL of your data provider, if not, you need to use the public IP address of your data provider. - -If you are using docker, you may need to use the public IP address of your data-provider or use the `--network host` option to reach a data-provider deployed on localhost. -More information on the [docker documentation](https://docs.docker.com/network/host/). +# Data Providers Quick Start + +## Creating a DebiAI Data Provider + +There are multiple ways to create a DebiAI Data Provider: + + + +### 1. DebiAI Data Provider Python Module (Recommended) + +The simplest way to create a Data Provider is by using the [DebiAI Data Provider Python module](https://github.com/debiai/easy-data-provider). This module allows you to define access methods and event handlers within a single Python class. + +### 2. Data Provider Templates + +To streamline Data Provider creation, we offer **quick-start templates**. Currently, templates are available for: + +- **Node.js**: [GitHub Repository](https://github.com/debiai/data-provider-nodejs-template) + +Want support for another language? [Let us know](https://github.com/debiai/data-provider-nodejs-template/issues/new). + +### 3. Custom Data Provider Implementation + +You can build your own Data Provider as long as it follows the [DebiAI Data Provider API](https://petstore.swagger.io/?url=https://raw.githubusercontent.com/debiai/data-provider-nodejs-template/main/data-provider-API.yaml). + +## Connecting a Data Provider to DebiAI + +After creating your Data Provider, you must configure DebiAI to access it: + + + +::: warning +**DebiAI must be able to access your Data Provider.** + +- If running locally, use `localhost` as the URL. +- If hosted externally, use the **public IP address**. +- When using **Docker**, you may need to use the public IP or `--network host` to access a provider deployed on `localhost`. + More details: [Docker documentation](https://docs.docker.com/network/host/). + ::: + +--- + +### 1. Connecting via the Dashboard + +You can add a Data Provider through the **DebiAI dashboard**: + +1. Click **Manage Data Providers** on the home page. +2. Click **New Data Provider**. +3. Fill in the following details: + - **Data Provider Name**: A unique name for identification. + - **Data Provider URL**: The endpoint where DebiAI can access the provider. +4. Click **Save**. If the provider is accessible and API-compliant, DebiAI will list the projects in the dashboard. + +![Add Data Provider](./data-provider-manager-data-provider-added.png) + +::: warning +**Dashboard-added providers are not persistent.** Restarting DebiAI will remove them. Use **environment variables** or a **configuration file** for persistence. ::: -### From the dashboard - -If you are using the DebiAI dashboard, you can add your data provider from the home page. First, click on the `Manage data providers` button on top of the page, the following panel will appear: +--- -![Add data provider](./data-provider-manager.png) +### 2. Connecting via Environment Variables -The `Python module Data Provider` is the default data provider that is used when you are using the [Debiai Python module](../pythonModule/README.md), you can [disable it](../pythonModule/quickStart.md#disabling-the-debiai-module-data-provider) if you don't need it. +For deployments, you can define environment variables to specify provider URLs. -To add a new **WEB data provider**, click on the `New data provider` button: - -![Add data provider](./data-provider-manager-new-data-provider.png) - -Then fill the form with the following information: - -- **Data provider name**: The name of your data provider. This name will be used to identify your data provider in DebiAI. -- **Data provider URL**: The URL of your data provider. This URL will be used by DebiAI to access your data provider. - -Once you have filled the form, click on the `save` button. If the data provider is accessible and is conform with the API, DebiAI will display the projects in the dashboard like any other projects. - -![Add data provider](./data-provider-manager-data-provider-added.png) - -::: warning -Restarting DebiAI will erase the data providers you have added from the dashboard. If you want to keep them, you will need to use the [environment variables](#environment-variables) or the [configuration file](#configuration-file). -::: - -### Environment variables - -If you are deploying DebiAI, you can set the environment variables `DEBIAI_WEB_DATA_PROVIDER_=` to tell DebiAI where to find your data-provider. +#### Example: ```bash -# Creation of the environment variables -export DEBIAI_WEB_DATA_PROVIDER_My-data-provider1=http://localhost:3000/debiai -export DEBIAI_WEB_DATA_PROVIDER_My-data-provider2=http://localhost:3010/ +export DEBIAI_WEB_DATA_PROVIDER_MyDataProvider1=http://localhost:3000/debiai +export DEBIAI_WEB_DATA_PROVIDER_MyDataProvider2=http://localhost:3010/ ``` +With Docker: + ```bash -# Example of use with docker run docker run -p 3000:3000 \ - -e DEBIAI_WEB_DATA_PROVIDER_My-data-provider1=http://localhost:3000/debiai \ - -e DEBIAI_WEB_DATA_PROVIDER_My-data-provider2=http://localhost:3010/ \ + -e DEBIAI_WEB_DATA_PROVIDER_MyDataProvider1=http://localhost:3000/debiai \ + -e DEBIAI_WEB_DATA_PROVIDER_MyDataProvider2=http://localhost:3010/ \ debiai/app ``` +With Docker Compose: + ```yaml -# Or with docker-compose version: "3.8" services: debiai: @@ -83,27 +129,27 @@ services: ports: - "3000:3000" environment: - # Data providers (DEBIAI_WEB_DATA_PROVIDER_=) - - DEBIAI_WEB_DATA_PROVIDER_My-data-provider1=http://localhost:3000/debiai\ - - DEBIAI_WEB_DATA_PROVIDER_My-data-provider2=http://localhost:3010/ + - DEBIAI_WEB_DATA_PROVIDER_MyDataProvider1=http://localhost:3000/debiai + - DEBIAI_WEB_DATA_PROVIDER_MyDataProvider2=http://localhost:3010/ ``` -You can see some examples in our DebiAI [`docker-compose-build.yml`](https://github.com/debiai/debiai/blob/main/docker-compose-build.yml) file. +For a full list fo environment variables, check the [docker-compose-build.yml](https://github.com/debiai/debiai/blob/main/docker-compose-build.yml) file. -### Configuration file +### 3. Connecting via Configuration File -You can also edit the [`debiai/debiaiServer/config/config.ini`](https://github.com/debiai/debiai/blob/main/debiaiServer/config/config.ini) file : +You can also configure providers in config.ini: +Example `(debiai/debiaiServer/config/config.ini)`: ```ini [DATA_PROVIDERS] -# List all the data providers you want to use -My-data-provider1 = http://localhost:3000/debiai/ -My-data-provider2 = http://localhost:3010/ +MyDataProvider1 = http://localhost:3000/debiai/ +MyDataProvider2 = http://localhost:3010/ ``` -You will then need to restart DebiAI if you are in development mode or to build a new docker image. Our [`Docker-compose-build`](https://github.com/debiai/debiai/blob/main/docker-compose-build.yml) file can help you with that. - -The environment variables take precedence over the configuration file. +After editing, restart DebiAI (or rebuild the Docker image if using containers). -If the data-provider is accessible and is conform with the API, DebiAI will display the projects in the dashboard like any other projects. +::: tip Priority Order: +Environment variables override the configuration file settings. +::: +If the provider is accessible and follows the API, DebiAI will list the projects in the dashboard. diff --git a/documentation/dataInsertion/pythonModule/README.md b/documentation/dataInsertion/pythonModule/README.md index 893eee3..6b66f78 100644 --- a/documentation/dataInsertion/pythonModule/README.md +++ b/documentation/dataInsertion/pythonModule/README.md @@ -4,13 +4,22 @@ The DebiAI Python module is an interface for DebiAI. You can use it directly in your Python project workflow for providing data to DebiAI. -## Features : -- Basic: - - Project creation - - Project data insertion - - Model metadata and model results insertion -- Advanced: - - Selection made with the dashboard samples recovery (see "Dashboard" for more information about selections) - - tf.dataset creation from the project selections *(beta)* +## Features + +- Project creation and deletion +- Project data insertion +- Model metadata and model results insertion +- Selection creation and retrieval [Quick start](./quickStart.md) + +## Disabling the DebiAI module Data Provider + +If you are using a [data provider](../dataProviders/) and you want to make make DebiAI read only, you may want to disable the DebiAI python module Data Provider. + +If you want to disable the DebiAI module, you can do it by adding the environment variable `DEBIAI_PYTHON_MODULE_DATA_PROVIDER_ENABLED` to `False` when starting DebiAI, or edit the `config.ini` file: + +```ini +[PYTHON_MODULE_DATA_PROVIDER] +enabled = False +``` diff --git a/documentation/dataInsertion/pythonModule/quickStart.md b/documentation/dataInsertion/pythonModule/quickStart.md index d80cae9..988289f 100644 --- a/documentation/dataInsertion/pythonModule/quickStart.md +++ b/documentation/dataInsertion/pythonModule/quickStart.md @@ -226,14 +226,3 @@ Now, both the data and the 2 models are ready to be analyzed with the Debiai das Note that the DebiAI Python module is in Alpha version, and is not yet stable, feel free to report any issue or suggestion. #### The next step is to [analyze your data with the DebiAI dashboard](../../dashboard/README.md). - -## Disabling the DebiAI module Data Provider - -If you are using a [WEB Pata Provider](../dataProviders/) and you want to make make DebiAI read only, you may want to disable the DebiAI python module Data Provider. - -If you want to disable the DebiAI module, you can do it by adding the environment variable `DEBIAI_PYTHON_MODULE_DATA_PROVIDER_ENABLED` to `False` when starting DebiAI, or edit the `config.ini` file: - -```ini -[PYTHON_MODULE_DATA_PROVIDER] -enabled = False -``` diff --git a/documentation/introduction/gettingStarted/installation/README.md b/documentation/introduction/gettingStarted/installation/README.md index 5ed3b6a..0d12d0e 100644 --- a/documentation/introduction/gettingStarted/installation/README.md +++ b/documentation/introduction/gettingStarted/installation/README.md @@ -5,67 +5,55 @@ DebiAI offers several installation methods: ## Debiai-gui python package -### Installation - -**Requirements :** +### Requirements: - [Python](https://www.python.org/downloads/) -Optional : create a virtual python environment: +### Virtual environment (optional) -``` +```bash python -m venv debiaivenv -``` -Activate the virtual python environment created, on Unix and macOS: - -```bash +# Activate the virtual python environment created: +# On Unix and macOS: source debiaivenv/bin/activate -``` - -Activate the virtual python environment created, on Windows PowerShell: - -``` +# On Windows PowerShell: debiaivenv\Scripts\Activate -``` - -Activate the virtual python environment created, on Windows terminal: - -``` +# On Windows terminal: debiaivenv\Scripts\activate ``` +### Installation -Install the python package with pip: +Install the `debiai-gui` python package with pip: ```bash pip install debiai-gui @@ -79,7 +67,18 @@ The following command will launch the debiai web application: debiai-gui start ``` -Debiai will be available on [http://localhost:3000/](http://localhost:3000/) +DebiAI needs a place to store the data, on startup it will ask you to provide a path to a folder where it can create a folder. + +Debiai will then be available on [http://localhost:3000/](http://localhost:3000/) + +**Parameters** +```bash +# Start DebiAI on a different port: +debiai-gui start --port 4000 + +# Display help: +debiai-gui --help +``` ## Official Docker image