Skip to main content

User Stage

Summary​

Supports user internal stage.

Motivation​

Databend only supports named internal stage:

CREATE STAGE @my_stage;
COPY INTO my_table FROM @my_stage;

However, named internal stages are complex to be used in some cases. Especially for users who only use stages to load data. By supporting the user stage, they can copy data more efficiently:

COPY INTO my_table from @~;

Guide-level explanation​

Databend will add support for user stage. Every sql user will have its own stage, which can be referred to by ~.

Users can use ~ everywhere like a named stage:

COPY INTO my_table FROM @~;
LIST @~;
PRESIGN @~/data.csv;
REMOVE @~ PATTERN = 'ontime.*';

User stage is the internal anonymous stage for sql user, so users can't:

  • create
  • drop
  • alter

And users can't set format options for user stage. They need to specify the format settings during COPY.

Reference-level explanation​

Databend now has two different StageType:

pub enum StageType {
    Internal,
    External,
}

Databend will generate a unique prefix for the internal stage, like stage/{stage_name}.

We will add two new stage types:

pub enum StageType {
    LegacyInternal,
    External,
    Internal,
    User,
}

StageType::Internal will deprecate StageType::LegacyInternal. Since this RFC, we will not create a new stage with StageType::LegacyInternal anymore.

The stage prefix rule will be:

  • LegacyInternal => stage/{stage_name}
  • External => specified location.
  • Internal => stage/internal/{stage_name}
  • User => stage/user/{user_name}

Notes: StageType::User will not be stored in metasrv and will constantly build in memory directly.

Drawbacks​

None.

Rationale and alternatives​

Preserve stage name prefix​

To simplify, we can preserve all stages prefixed by bend_internal_. Users can't create and drop stages with this prefix.

By adding this limitation, we can implement the user stage easier. Every time user tries to access their own user stage, we will expand to bend_internal_user_<user_name>.

Take user root as an example:

COPY INTO my_table FROM @~;

will be transformed into:

COPY INTO my_table FROM @bend_internal_user_root;

Users can only access their user stage by @~. Visit @bend_internal_user_root will always return an error.

Create stage with UUID in metasrv​

We can create a stage with UUID for first-time users to access their user stage.

Prior art​

None

Unresolved questions​

None

Future possibilities​

Table Stage​

We can introduce the table stage like snowflake does:

COPY INTO my_table FROM @#my_table;

Cleanup while drop users​

Users' stage should be purged while dropping user.

Garbage Collection for user stage​

We can support garbage collection for user stages so that obsoleted files can be removed.