-
Notifications
You must be signed in to change notification settings - Fork 3.2k
feat(tinybird): added tinybird block #2781
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
lakeesiv
wants to merge
33
commits into
simstudioai:staging
Choose a base branch
from
lakeesiv:lakees/tinybird
base: staging
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+666
−2
Open
Changes from all commits
Commits
Show all changes
33 commits
Select commit
Hold shift + click to select a range
eb07a08
v0.5.35: helm updates, copilot improvements, 404 for docs, salesforce…
icecrasher321 4d1a9a3
v0.5.36: hitl improvements, opengraph, slack fixes, one-click unsubsc…
waleedlatif1 4431a1a
fix(helm): add custom egress rules to realtime network policy (#2481)
Lutherwaves 3e697d9
v0.5.37: redaction utils consolidation, logs updates, autoconnect imp…
waleedlatif1 4827866
v0.5.38: snap to grid, copilot ux improvements, billing line items
waleedlatif1 0f4ec96
v0.5.39: notion, workflow variables fixes
waleedlatif1 3d9d9cb
v0.5.40: supabase ops to allow non-public schemas, jira uuid
icecrasher321 e12dd20
v0.5.41: memory fixes, copilot improvements, knowledgebase improvemen…
icecrasher321 57e4b49
v0.5.42: fix memory migration
icecrasher321 b304233
v0.5.43: export logs, circleback, grain, vertex, code hygiene, schedu…
waleedlatif1 b6ba3b5
v0.5.44: keyboard shortcuts, autolayout, light mode, byok, testing im…
waleedlatif1 dd3209a
v0.5.45: light mode fixes, realtime usage indicator, docker build imp…
waleedlatif1 f895bf4
v0.5.46: build improvements, greptile, light mode improvements
waleedlatif1 54ab82c
v0.5.47: deploy workflow as mcp, kb chunks tokenizer, UI improvements…
waleedlatif1 eb5d1f3
v0.5.48: copy-paste workflow blocks, docs updates, mcp tool fixes
waleedlatif1 3792bdd
v0.5.49: hitl improvements, new email styles, imap trigger, logs cont…
waleedlatif1 585f5e3
v0.5.50: import improvements, ui upgrades, kb styling and performance…
waleedlatif1 4fbec0a
v0.5.51: triggers, kb, condition block improvements, supabase and gra…
waleedlatif1 bfb6fff
v0.5.52: new port-based router block, combobox expression and variabl…
waleedlatif1 f5ab7f2
v0.5.53: hotkey improvements, added redis fallback, fixes for workflo…
waleedlatif1 13a6e6c
v0.5.54: seo, model blacklist, helm chart updates, fireflies integrat…
waleedlatif1 f415e5e
v0.5.55: polling groups, bedrock provider, devcontainer fixes, workfl…
waleedlatif1 be578e2
v0.5.56: batch operations, access control and permission groups, bill…
waleedlatif1 7ffc11a
v0.5.57: subagents, context menu improvements, bug fixes
icecrasher321 3907427
updates
lakeesiv 217d4a9
Update apps/sim/components/icons.tsx
lakeesiv 186c54f
code tag
lakeesiv a1ef5ab
updates
lakeesiv fc4ecc8
Merge branch 'lakees/tinybird' of https://github.com/lakeesiv/sim int…
lakeesiv 0f686c8
Update apps/sim/blocks/blocks/tinybird.ts
lakeesiv 254f799
Update apps/sim/tools/tinybird/query.ts
lakeesiv c9b0c2c
docs
lakeesiv c979c70
Merge branch 'lakees/tinybird' of https://github.com/lakeesiv/sim int…
lakeesiv File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -102,6 +102,7 @@ | |
| "tavily", | ||
| "telegram", | ||
| "thinking", | ||
| "tinybird", | ||
| "translate", | ||
| "trello", | ||
| "tts", | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| --- | ||
| title: Tinybird | ||
| description: Send events and query data with Tinybird | ||
| --- | ||
|
|
||
| import { BlockInfoCard } from "@/components/ui/block-info-card" | ||
|
|
||
| <BlockInfoCard | ||
| type="tinybird" | ||
| color="#2EF598" | ||
| /> | ||
|
|
||
| ## Usage Instructions | ||
|
|
||
| Interact with Tinybird using the Events API to stream JSON or NDJSON events, or use the Query API to execute SQL queries against Pipes and Data Sources. | ||
|
|
||
|
|
||
|
|
||
| ## Tools | ||
|
|
||
| ### `tinybird_events` | ||
|
|
||
| Send events to a Tinybird Data Source using the Events API. Supports JSON and NDJSON formats with optional gzip compression. | ||
|
|
||
| #### Input | ||
|
|
||
| | Parameter | Type | Required | Description | | ||
| | --------- | ---- | -------- | ----------- | | ||
| | `base_url` | string | Yes | Tinybird API base URL \(e.g., https://api.tinybird.co or https://api.us-east.tinybird.co\) | | ||
| | `datasource` | string | Yes | Name of the Tinybird Data Source to send events to | | ||
| | `data` | string | Yes | Data to send as NDJSON \(newline-delimited JSON\) or JSON string. Each event should be a valid JSON object. | | ||
| | `wait` | boolean | No | Wait for database acknowledgment before responding. Enables safer retries but introduces latency. Defaults to false. | | ||
| | `format` | string | No | Format of the events data: "ndjson" \(default\) or "json" | | ||
| | `compression` | string | No | Compression format: "none" \(default\) or "gzip" | | ||
| | `token` | string | Yes | Tinybird API Token with DATASOURCE:APPEND or DATASOURCE:CREATE scope | | ||
|
|
||
| #### Output | ||
|
|
||
| | Parameter | Type | Description | | ||
| | --------- | ---- | ----------- | | ||
| | `successful_rows` | number | Number of rows successfully ingested | | ||
| | `quarantined_rows` | number | Number of rows quarantined \(failed validation\) | | ||
|
|
||
| ### `tinybird_query` | ||
|
|
||
| Execute SQL queries against Tinybird Pipes and Data Sources using the Query API. | ||
|
|
||
| #### Input | ||
|
|
||
| | Parameter | Type | Required | Description | | ||
| | --------- | ---- | -------- | ----------- | | ||
| | `base_url` | string | Yes | Tinybird API base URL \(e.g., https://api.tinybird.co\) | | ||
| | `query` | string | Yes | SQL query to execute. Specify your desired output format \(e.g., FORMAT JSON, FORMAT CSV, FORMAT TSV\). JSON format provides structured data, while other formats return raw text. | | ||
| | `pipeline` | string | No | Optional pipe name. When provided, enables SELECT * FROM _ syntax | | ||
| | `token` | string | Yes | Tinybird API Token with PIPE:READ scope | | ||
|
|
||
| #### Output | ||
|
|
||
| | Parameter | Type | Description | | ||
| | --------- | ---- | ----------- | | ||
| | `data` | json | Query result data. For FORMAT JSON: array of objects. For other formats \(CSV, TSV, etc.\): raw text string. | | ||
| | `rows` | number | Number of rows returned \(only available with FORMAT JSON\) | | ||
| | `statistics` | json | Query execution statistics - elapsed time, rows read, bytes read \(only available with FORMAT JSON\) | | ||
|
|
||
|
|
||
|
|
||
| ## Notes | ||
|
|
||
| - Category: `tools` | ||
| - Type: `tinybird` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,206 @@ | ||
| import { TinybirdIcon } from '@/components/icons' | ||
| import type { BlockConfig } from '@/blocks/types' | ||
| import { AuthMode } from '@/blocks/types' | ||
| import type { TinybirdResponse } from '@/tools/tinybird/types' | ||
|
|
||
| export const TinybirdBlock: BlockConfig<TinybirdResponse> = { | ||
| type: 'tinybird', | ||
| name: 'Tinybird', | ||
| description: 'Send events and query data with Tinybird', | ||
| authMode: AuthMode.ApiKey, | ||
| longDescription: | ||
| 'Interact with Tinybird using the Events API to stream JSON or NDJSON events, or use the Query API to execute SQL queries against Pipes and Data Sources.', | ||
| docsLink: 'https://www.tinybird.co/docs/api-reference', | ||
| category: 'tools', | ||
| bgColor: '#2EF598', | ||
| icon: TinybirdIcon, | ||
| subBlocks: [ | ||
| { | ||
| id: 'operation', | ||
| title: 'Operation', | ||
| type: 'dropdown', | ||
| options: [ | ||
| { label: 'Send Events', id: 'tinybird_events' }, | ||
| { label: 'Query', id: 'tinybird_query' }, | ||
| ], | ||
| value: () => 'tinybird_events', | ||
| }, | ||
| { | ||
| id: 'base_url', | ||
| title: 'Base URL', | ||
| type: 'short-input', | ||
| placeholder: 'https://api.tinybird.co', | ||
| required: true, | ||
| }, | ||
| { | ||
| id: 'token', | ||
| title: 'API Token', | ||
| type: 'short-input', | ||
| placeholder: 'Enter your Tinybird API token', | ||
| password: true, | ||
| required: true, | ||
| }, | ||
| // Send Events operation inputs | ||
| { | ||
| id: 'datasource', | ||
| title: 'Data Source', | ||
| type: 'short-input', | ||
| placeholder: 'my_events_datasource', | ||
| condition: { field: 'operation', value: 'tinybird_events' }, | ||
| required: true, | ||
| }, | ||
| { | ||
| id: 'data', | ||
| title: 'Data', | ||
| type: 'code', | ||
| placeholder: | ||
| '{"event": "click", "timestamp": "2024-01-01T12:00:00Z"}\n{"event": "view", "timestamp": "2024-01-01T12:00:01Z"}', | ||
| condition: { field: 'operation', value: 'tinybird_events' }, | ||
| required: true, | ||
| }, | ||
| { | ||
| id: 'format', | ||
| title: 'Format', | ||
| type: 'dropdown', | ||
| options: [ | ||
| { label: 'NDJSON (Newline-delimited JSON)', id: 'ndjson' }, | ||
| { label: 'JSON', id: 'json' }, | ||
| ], | ||
| value: () => 'ndjson', | ||
| condition: { field: 'operation', value: 'tinybird_events' }, | ||
| }, | ||
| { | ||
| id: 'compression', | ||
| title: 'Compression', | ||
| type: 'dropdown', | ||
| options: [ | ||
| { label: 'None', id: 'none' }, | ||
| { label: 'Gzip', id: 'gzip' }, | ||
| ], | ||
| value: () => 'none', | ||
| mode: 'advanced', | ||
| condition: { field: 'operation', value: 'tinybird_events' }, | ||
| }, | ||
| { | ||
| id: 'wait', | ||
| title: 'Wait for Acknowledgment', | ||
| type: 'switch', | ||
| value: () => 'false', | ||
| mode: 'advanced', | ||
| condition: { field: 'operation', value: 'tinybird_events' }, | ||
| }, | ||
| // Query operation inputs | ||
| { | ||
| id: 'query', | ||
| title: 'SQL Query', | ||
| type: 'code', | ||
| placeholder: 'SELECT * FROM my_pipe FORMAT JSON\nOR\nSELECT * FROM my_pipe FORMAT CSV', | ||
| condition: { field: 'operation', value: 'tinybird_query' }, | ||
| required: true, | ||
| }, | ||
| { | ||
| id: 'pipeline', | ||
| title: 'Pipeline Name', | ||
| type: 'short-input', | ||
| placeholder: 'my_pipe (optional)', | ||
| condition: { field: 'operation', value: 'tinybird_query' }, | ||
| }, | ||
| ], | ||
| tools: { | ||
| access: ['tinybird_events', 'tinybird_query'], | ||
| config: { | ||
| tool: (params) => params.operation || 'tinybird_events', | ||
| params: (params) => { | ||
| const operation = params.operation || 'tinybird_events' | ||
| const result: Record<string, any> = { | ||
| base_url: params.base_url, | ||
| token: params.token, | ||
| } | ||
|
|
||
| if (operation === 'tinybird_events') { | ||
| // Send Events operation | ||
| if (!params.datasource) { | ||
| throw new Error('Data Source is required for Send Events operation') | ||
| } | ||
| if (!params.data) { | ||
| throw new Error('Data is required for Send Events operation') | ||
| } | ||
|
|
||
| result.datasource = params.datasource | ||
| result.data = params.data | ||
| result.format = params.format || 'ndjson' | ||
| result.compression = params.compression || 'none' | ||
|
|
||
| // Convert wait from string to boolean | ||
| // Convert wait from string to boolean | ||
| if (params.wait !== undefined) { | ||
| const waitValue = typeof params.wait === 'string' ? params.wait.toLowerCase() : params.wait | ||
| result.wait = waitValue === 'true' || waitValue === true | ||
| } | ||
| } else if (operation === 'tinybird_query') { | ||
| // Query operation | ||
| if (!params.query) { | ||
| throw new Error('SQL Query is required for Query operation') | ||
| } | ||
|
|
||
| result.query = params.query | ||
| if (params.pipeline) { | ||
| result.pipeline = params.pipeline | ||
| } | ||
| } | ||
|
|
||
| return result | ||
| }, | ||
| }, | ||
| }, | ||
| inputs: { | ||
| operation: { type: 'string', description: 'Operation to perform' }, | ||
| base_url: { type: 'string', description: 'Tinybird API base URL' }, | ||
| // Send Events inputs | ||
| datasource: { | ||
| type: 'string', | ||
| description: 'Name of the Tinybird Data Source', | ||
| }, | ||
| data: { | ||
| type: 'string', | ||
| description: 'Data to send as JSON or NDJSON string', | ||
| }, | ||
| wait: { type: 'boolean', description: 'Wait for database acknowledgment' }, | ||
| format: { | ||
| type: 'string', | ||
| description: 'Format of the events (ndjson or json)', | ||
| }, | ||
| compression: { | ||
| type: 'string', | ||
| description: 'Compression format (none or gzip)', | ||
| }, | ||
| // Query inputs | ||
| query: { type: 'string', description: 'SQL query to execute' }, | ||
| pipeline: { type: 'string', description: 'Optional pipeline name' }, | ||
| // Common | ||
| token: { type: 'string', description: 'Tinybird API Token' }, | ||
| }, | ||
| outputs: { | ||
| // Send Events outputs | ||
| successful_rows: { | ||
| type: 'number', | ||
| description: 'Number of rows successfully ingested', | ||
| }, | ||
| quarantined_rows: { | ||
| type: 'number', | ||
| description: 'Number of rows quarantined (failed validation)', | ||
| }, | ||
| // Query outputs | ||
| data: { | ||
| type: 'json', | ||
| description: | ||
| 'Query result data. FORMAT JSON: array of objects. Other formats (CSV, TSV, etc.): raw text string.', | ||
| }, | ||
| rows: { type: 'number', description: 'Number of rows returned (only with FORMAT JSON)' }, | ||
| statistics: { | ||
| type: 'json', | ||
| description: | ||
| 'Query execution statistics - elapsed time, rows read, bytes read (only with FORMAT JSON)', | ||
| }, | ||
| }, | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.