DolphinScheduler Shell and native SQL tasks

Routes and version

The reference version is DolphinScheduler 3.4.2. Real Shell tasks launched Python/PyMySQL for multiple scenarios; separate native SQL/JDBC datasource tasks verified COUNT/SUM. The Shell type matrix is not full JDBC type certification, and the aggregate preflight does not establish heavy scheduler behavior or HA.

Deploy a test instance following the DolphinScheduler installation guide, with a tenant, worker group and execution directory. Complete the shared fixtures. The worker must reach the database, not just the API server.

Shell: script, environment and exit status

Install mo_common.py and ds_check.py below in a dedicated worker directory. Inject MO_* through managed environment configuration rather than embedding passwords in visible Shell commands. Create a Shell task running .venv/bin/python ds_check.py, with a timeout and bounded retries. After SUCCESS, inspect its JSON readback and destination values.

import json
from decimal import Decimal
from mo_common import connect

with connect() as d:
    with d.cursor() as c:
        c.execute("SELECT id,amount,HEX(payload),label,rev FROM lake_demo WHERE id IN (2,9007199254740993) ORDER BY id")
        rows = c.fetchall()
assert len(rows) == 2
assert rows[0] == (2,None,"","",0)
assert rows[1] == (9007199254740993,Decimal("12345678901234567890.123456789012345678"),"00FF80","中文😀",0)
print(json.dumps(rows,ensure_ascii=False,default=str))

Native SQL/JDBC: configure separately

Choose MYSQL in datasource management, map the shared host/port/database/user/password and install a MySQL JDBC driver compatible with the DS release. Confirm com.mysql.cj.jdbc.Driver loads on the worker’s datasource path. Test the datasource connection, then create a SQL task using it; a connection test is not task execution.

Initialize the audit table in the dedicated database. The complete example includes DDL and a query for a client preflight; configure the SQL task to execute only the INSERT, with the non-query/update SQL type. The fixed ds-demo key makes retries write the same statistics. It is a demonstration, not a general event exactly-once protocol.

CREATE DATABASE IF NOT EXISTS doc_lake;
USE doc_lake;
CREATE TABLE IF NOT EXISTS ds_demo_audit (run_key VARCHAR(64) PRIMARY KEY, rows_seen BIGINT, sum_rev BIGINT);
CREATE TABLE IF NOT EXISTS lake_demo (id BIGINT PRIMARY KEY, amount DECIMAL(38,18), payload VARBINARY(256), label VARCHAR(256), rev BIGINT);
INSERT INTO ds_demo_audit
SELECT 'ds-demo', COUNT(*), COALESCE(SUM(rev),0) FROM lake_demo
ON DUPLICATE KEY UPDATE rows_seen=VALUES(rows_seen),sum_rev=VALUES(sum_rev);
SELECT * FROM ds_demo_audit WHERE run_key='ds-demo';

Release, recovery and troubleshooting

Release the workflow ONLINE and trigger a run. Record workflow/task instances, worker, retry count and executed SQL. For a SQL task failure, retain the JDBC exception and generated statement. For Shell failures, check the tenant OS user, working directory, Python environment, network and transaction errors separately.

After a worker restart, verify committed audit rows as well as scheduler state. Implement business deletes, schema evolution and batch commits explicitly in the task. If the source changes during retries, overwriting a fixed run_key records the latest observation rather than an immutable snapshot; production batches need a fixed input watermark. Qualify worker resources, connection limits and master/worker failure recovery under concurrency.